Data compensation method and device, equipment and storage medium

Abstract

The invention discloses a data compensation method, which is used for solving the problem that the use consistency of downstream data is influenced due to the fact that the existing data compensation scheme is easy to cause the difference of compensated data among different subscribers. The method comprises the steps of performing data verification on subscription terminal data and data in a source end database through a first subscription terminal, and determining abnormal data; searching compensation data corresponding to the abnormal data in a source end database according to the abnormal data; packaging the compensation data according to a preset message format, and pushing the compensation data to a compensation theme corresponding to the message middleware; and issuing a data compensation notification carrying a compensation theme to each subscription terminal in the downstream subscription terminal group, so that each subscription terminal in the downstream subscription terminal group searches compensation data in the message-oriented middleware according to the compensation theme, and performs data compensation.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data compensation method, apparatus, device, and storage medium. Background Technology

[0002] In streaming data processing systems, especially in data subscription and distribution scenarios based on message middleware (such as Kafka), multiple downstream subscribers typically need to verify and compensate the data they receive to ensure data integrity and consistency.

[0003] Currently, in existing technologies, the common method for compensating streaming data involves each subscriber independently performing data verification and compensation operations. When each subscriber discovers potential data omissions or errors, it automatically connects to the source database, queries for the correct data, and makes the corrections.

[0004] However, this approach has certain limitations because it relies on each subscriber processing the data independently. On the one hand, each subscriber needs to establish a connection with the source database and perform repetitive data queries and comparisons, leading to an increase in the number of database connections and potentially putting pressure on database performance. On the other hand, since subscribers may have inconsistent data verification strategies, comparison granularity, or processing logic, the compensated data may differ between different subscribers, affecting the consistency of downstream data usage.

[0005] Therefore, there is an urgent need for a unified and efficient data compensation method that can perform streaming data verification and compensation, and reduce the load on the source system while ensuring data consistency. Summary of the Invention

[0006] This application provides a data compensation method to solve the problem that existing data compensation schemes require each subscriber to perform data verification with the end database separately. Since the data verification strategies, comparison granularity, or processing logic of each subscriber may be inconsistent, it is easy to cause differences in the compensated data between different subscribers, which affects the consistency of downstream data use.

[0007] This application also provides a data compensation device to solve the problem that existing data compensation schemes require each subscriber to perform data verification with the end database separately. Since the data verification strategies, comparison granularity, or processing logic of each subscriber may be inconsistent, it is easy to cause differences in the compensated data between different subscribers, which affects the consistency of downstream data use.

[0008] This application also provides a data compensation device to solve the problem that existing data compensation schemes require each subscriber to perform data verification with the end database separately. Since each subscriber may have different data verification strategies, comparison granularity, or processing logic, the compensated data may differ between different subscribers, affecting the consistency of downstream data use.

[0009] This application also provides a computer-readable storage medium to solve the problem that existing data compensation schemes require each subscriber to perform data verification with the end database separately. Since each subscriber may have different data verification strategies, comparison granularity, or processing logic, the compensated data may differ between different subscribers, affecting the consistency of downstream data use.

[0010] The embodiments of this application adopt the following technical solutions: A data compensation method includes: performing data verification between a first subscription terminal and data in a source database to identify abnormal data, wherein the first subscription terminal is any subscription terminal in a downstream subscription terminal group of the source database; searching for compensation data corresponding to the abnormal data in the source database based on the abnormal data; encapsulating the compensation data according to a preset message format and pushing it to a corresponding compensation topic in a message middleware, wherein the compensation topic is set according to the information of each data table in the source database and is used to distinguish compensation data from different sources; and sending a data compensation notification carrying the compensation topic to each subscription terminal in the downstream subscription terminal group, so that each subscription terminal in the downstream subscription terminal group can search for compensation data in the message middleware according to the compensation topic and perform data compensation.

[0011] A data compensation device includes: an abnormal data determination unit, configured to verify data between a subscription terminal and data in a source database using a first subscription terminal to determine abnormal data, wherein the first subscription terminal is any subscription terminal in a downstream subscription terminal group of the source database; a compensation data lookup unit, configured to look up compensation data corresponding to the abnormal data in the source database; a message push unit, configured to encapsulate the compensation data according to a preset message format and push it to a corresponding compensation topic in a message middleware, wherein the compensation topic is set according to information from each data table in the source database to distinguish compensation data from different sources; and a compensation unit, configured to send a data compensation notification carrying a compensation topic to each subscription terminal in the downstream subscription terminal group, so that each subscription terminal in the downstream subscription terminal group can look up compensation data in the message middleware according to the compensation topic and perform data compensation.

[0012] A data compensation device, comprising: The processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the following operations: verifying data between a subscription terminal and data in a source database using a first subscription terminal to identify abnormal data, wherein the first subscription terminal is any subscription terminal in the downstream subscription terminal group of the source database; searching for compensation data corresponding to the abnormal data in the source database based on the abnormal data; encapsulating the compensation data according to a preset message format and pushing it to a corresponding compensation topic in a message middleware, wherein the compensation topic is set based on information from various data tables in the source database to distinguish compensation data from different sources; and sending a data compensation notification carrying the compensation topic to each subscription terminal in the downstream subscription terminal group, so that each subscription terminal in the downstream subscription terminal group can search for compensation data in the message middleware based on the compensation topic and perform data compensation.

[0013] A computer-readable storage medium stores one or more programs that, when executed by an electronic device including multiple applications, cause the electronic device to perform the following operations: verifying data between a first subscription terminal and data in a source database using a first subscription terminal to identify abnormal data, wherein the first subscription terminal is any subscription terminal in a downstream subscription terminal group of the source database; searching for compensation data corresponding to the abnormal data in the source database based on the abnormal data; encapsulating the compensation data according to a preset message format and pushing it to a corresponding compensation topic in a message middleware, wherein the compensation topic is set based on information from various data tables in the source database to distinguish compensation data from different sources; and sending a data compensation notification carrying the compensation topic to each subscription terminal in the downstream subscription terminal group, so that each subscription terminal in the downstream subscription terminal group searches for compensation data in the message middleware based on the compensation topic and performs data compensation.

[0014] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects: Using the data compensation method provided in this application embodiment, when data verification and compensation are required, a first subscription terminal can verify the data of the subscription terminal and the data in the source database to identify abnormal data. The first subscription terminal is any subscription terminal in the downstream subscription terminal group of the source database. Based on the abnormal data, compensation data corresponding to the abnormal data is searched in the source database. The compensation data is encapsulated according to a preset message format and pushed to the corresponding compensation topic in the message middleware. The compensation topic is set according to the information of each data table in the source database and is used to distinguish compensation data from different sources. A data compensation notification carrying the compensation topic is sent to each subscription terminal in the downstream subscription terminal group, so that each subscription terminal in the downstream subscription terminal group can search for compensation data in the message middleware according to the compensation topic and perform data compensation. The data compensation method provided in this application has several advantages. First, by having a single subscription terminal (the first subscription terminal) centrally execute data verification and compensation data query operations, the database connections and query requests that originally required all downstream subscription terminals to establish separately are merged into one. This significantly reduces the number of concurrent database connections and query load, effectively avoiding database performance degradation or service congestion caused by multiple terminals accessing the system simultaneously. This improves the stability and reliability of the entire data flow system. Furthermore, by distributing the acquisition of verification and compensation data to a single terminal, other downstream subscription terminals do not need to repeatedly execute the same data comparison and database query logic. This avoids the redundant consumption of computing resources, network bandwidth, and storage space, reducing the overall operating cost of the system and improving the overall efficiency of the data processing link. Second, all downstream subscription terminals obtain compensation data provided by the first subscription terminal from the same compensation topic in the message middleware. This data is uniformly verified and formatted according to standards, fundamentally eliminating data differences caused by inconsistencies in verification rules, processing logic, or data sources among different terminals. This ensures a high degree of consistency in the final data content of all subscription terminals, providing an accurate and reliable data foundation for downstream services. Finally, by creating independent compensation topics for different data tables, isolation and granular management of compensation data were achieved. Downstream subscription terminals can subscribe to specific compensation topics according to their own needs, making the compensation process more flexible. At the same time, the centralized processing method also facilitates unified monitoring, scheduling, and maintenance of the data compensation process, improving the manageability and maintainability of the system. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1This is a schematic diagram illustrating a specific process of a data compensation method provided in an embodiment of this application; Figure 2 This application provides a schematic diagram illustrating a specific process for constructing a thought chain. Figure 3 This is a schematic diagram of the specific structure of a data compensation device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the specific structure of a data compensation device provided in an embodiment of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0018] This application provides a data compensation method to address the problem that existing data compensation schemes require each subscriber to perform data verification with the end database. Since the data verification strategies, comparison granularity, or processing logic of each subscriber may be inconsistent, it is easy for the compensated data to differ between different subscribers, affecting the consistency of downstream data use.

[0019] The execution subject of the data compensation method provided in this application embodiment may be, but is not limited to, at least one of a banking business server, an insurance business server, a bank database, and a data compensation server; in addition, the execution subject of the method may also be the system or application (APP) itself running on these servers.

[0020] For ease of description, the following description uses a data compensation system as the execution subject of this method as an example to introduce its implementation. It should be understood that using a data compensation system as the execution subject is merely an illustrative example and should not be construed as a limitation of the method.

[0021] This application example uses a system architecture where the source database is a MySQL relational database, the message middleware is Apache Kafka, and the downstream subscription terminal group contains N subscription terminals (subscription terminal 1, subscription terminal 2, ..., subscription terminal N), as shown below. Figure 1 The method of the present invention will be described below.

[0022] Based on the above system architecture, a schematic diagram of the specific implementation process of the data compensation method provided in this application is shown below. Figure 2 As shown, the main steps include the following: Step 11: Verify the data on the subscription terminal and the data in the source database using the first subscription terminal to identify abnormal data; Wherein, the first subscription terminal is any subscription terminal in the downstream subscription terminal group of the source database. Specifically, the first subscription terminal can be designated or elected from the downstream subscription terminal group. In one embodiment, the first subscription terminal can be subscription terminal 1 in the downstream subscription terminal group.

[0023] In this embodiment of the application, the data compensation system can specifically determine abnormal data by the following method: according to a preset verification period, comparing the data of the subscription terminal with the data in the source database with all fields, and determining the data with inconsistent field values ​​as abnormal data.

[0024] Among them, the T+1 mode can be used as the preset verification period. By configuring a timed task in the data verification module of the subscription terminal 1, the verification period is set to 2:00 am every day (T+1 mode). The verification strategy of this task is to compare the incremental data generated by the specified data table on the previous natural day (T day).

[0025] Specifically, subscription terminal 1 can parse a snapshot of all data from the previous natural day (day T) from the Kafka original topic db1-table1 that it subscribes to and consumes. By establishing a connection with the source database, it can execute the corresponding SQL query statement (e.g., SELECT * FROM table1 WHERE update_time BETWEEN 'day T 00:00:00' AND 'day T 23:59:59') to obtain the official snapshot of the incremental data for day T in the source database.

[0026] Subscription terminal 1 performs a full-field comparison (i.e., compares all field values ​​for each record) on the data snapshots of the two data sources through the verification module. For example, if subscription terminal 1 finds a record with primary key (column_pkey) NO11110000, and its timestamp in its own subscription data is 2025-01-01 10:30:15.700000, while the value of this field in the source database is 2025-01-01 10:30:15.007000, then this data can be identified as abnormal data. Subscription terminal 1 can then mark all records with similar inconsistencies in field values ​​as abnormal data, record their primary key information, and thus generate an abnormal data list.

[0027] Step 12: Based on the abnormal data obtained by executing Step 11, search for the compensation data corresponding to the abnormal data in the source database; In one implementation, the subscription terminal 1 can use the primary key of the abnormal data to query and obtain complete compensation data from the source database.

[0028] Specifically, in this embodiment, the subscription terminal 1 can traverse each record in the abnormal data list generated by executing step 11. For each abnormal data record, it constructs a precise query SQL statement using its primary key (e.g., column_pkey = 'NO11110000'), for example: SELECT * FROM table1 WHERE column_pkey = 'NO11110000'. Through the established database connection, it executes the SQL statement to obtain the complete and correct latest version data of the record from the source database as compensation data.

[0029] Step 13: Encapsulate the compensation data obtained by executing Step 12 according to the preset message format and push it to the corresponding compensation topic in the message middleware; The compensation theme is set based on the information in each data table of the source database and is used to distinguish compensation data from different sources.

[0030] Specifically, in this embodiment, the subscription terminal 1 can encapsulate the queried compensation data in the same preset message format (e.g., JSON format) as the original Kafka topic db1-table1, and create a new compensation topic on Kafka through the Kafka management tool or API. The topic name is generated according to the rule "{database name}-{table name}-fix", i.e., db1-table1-fix. All the encapsulated compensation data messages are then pushed to the db1-table1-fix topic in Kafka in sequence.

[0031] Step 14: Send a data compensation notification carrying a compensation topic to each subscription terminal in the downstream subscription terminal group, so that each subscription terminal in the downstream subscription terminal group can find the compensation data in the message middleware according to the compensation topic and perform data compensation.

[0032] In one implementation, the data compensation system can authorize each subscription terminal in the downstream subscription terminal group to access the compensation topic, determine the start time node of the data compensation, generate a data compensation notification based on the start time node, and send the data compensation notification to each subscription terminal in the downstream subscription terminal group.

[0033] Specifically, the data compensation system can call the Access Control List (ACL) interface of the message middleware Kafka to authorize the consumer group ID of each subscription terminal in the downstream subscription terminal group to read the compensation topic db1-table1-fix.

[0034] Simultaneously, the data compensation system can determine the start time T of the data compensation. T is the timestamp of the earliest message in the compensation topic db1-table1-fix (or the creation time of the compensation topic db1-table1-fix). The data compensation system then generates and sends out a data compensation notification through an internal messaging system (such as email, API calls, or a configuration center). The notification content must include at least: the name of the compensation topic to be subscribed to (db1-table1-fix) and the start time T.

[0035] Upon receiving the data compensation notification, each subscription terminal in the downstream subscription terminal group will perform data compensation processing. The following section uses subscription terminal 2 as an example to introduce the specific data compensation scheme for the subscription terminal: After receiving the data compensation notification, subscription terminal 2's consumer program can first determine its original subscription time node T1 corresponding to its current consumption offset in the original data topic db1-table1 by querying Kafka. Then, based on the original subscription time node T1 and the start time node T, it determines the specific data compensation plan, including: Option 1: When the original subscription time node T1 is less than the start time node T, it means that the consumption progress of subscription terminal 2 is later than the time when the compensation data was generated, and the erroneous data has not yet been consumed. In this case, subscription terminal 2 can switch the consumption group (with the group-id remaining unchanged) from the original topic db1-table1 to the compensation topic db1-table1-fix for consumption. After consuming all messages in the compensation topic (or listening until the consumption lag is 0), it can switch back to the original topic db1-table1 to continue consumption. This process ensures that the compensation data is consumed preferentially and completely.

[0036] Option 2: When the original subscription time node T1 is greater than the start time node T, it indicates that the subscription terminal 2 may have consumed some erroneous data. In this case, we should first switch to the compensation topic db1-table1-fix and consume all messages. Before switching back to the original topic, a callback operation needs to be performed: calculate a second time node T2 (T2=T - 1) based on the start time node T and the preset duration (e.g., 1 hour) in the data compensation notification. Then, call the Kafka API to reset the offset of the consumer group on the original topic db1-table1 to the position corresponding to time point T2. After that, switch back to the original topic and start consuming again from T2 to ensure that any erroneous data that may have been consumed is covered and to ensure the final data consistency.

[0037] After all downstream subscription terminals confirm the completion of data compensation, the data compensation system can delete the compensation topic db1-table1-fix in Kafka, release resources, and the data compensation process ends.

[0038] Using the data compensation method provided in this application embodiment, when data verification and compensation are required, a first subscription terminal can verify the data of the subscription terminal and the data in the source database to identify abnormal data. The first subscription terminal is any subscription terminal in the downstream subscription terminal group of the source database. Based on the abnormal data, compensation data corresponding to the abnormal data is searched in the source database. The compensation data is encapsulated according to a preset message format and pushed to the corresponding compensation topic in the message middleware. The compensation topic is set according to the information of each data table in the source database and is used to distinguish compensation data from different sources. A data compensation notification carrying the compensation topic is sent to each subscription terminal in the downstream subscription terminal group, so that each subscription terminal in the downstream subscription terminal group can search for compensation data in the message middleware according to the compensation topic and perform data compensation. The data compensation method provided in this application has several advantages. First, by having a single subscription terminal (the first subscription terminal) centrally execute data verification and compensation data query operations, the database connections and query requests that originally required all downstream subscription terminals to establish separately are merged into one. This significantly reduces the number of concurrent database connections and query load, effectively avoiding database performance degradation or service congestion caused by multiple terminals accessing the system simultaneously. This improves the stability and reliability of the entire data flow system. Furthermore, by distributing the acquisition of verification and compensation data to a single terminal, other downstream subscription terminals do not need to repeatedly execute the same data comparison and database query logic. This avoids the redundant consumption of computing resources, network bandwidth, and storage space, reducing the overall operating cost of the system and improving the overall efficiency of the data processing link. Second, all downstream subscription terminals obtain compensation data provided by the first subscription terminal from the same compensation topic in the message middleware. This data is uniformly verified and formatted according to standards, fundamentally eliminating data differences caused by inconsistencies in verification rules, processing logic, or data sources among different terminals. This ensures a high degree of consistency in the final data content of all subscription terminals, providing an accurate and reliable data foundation for downstream services. Finally, by creating independent compensation topics for different data tables, isolation and granular management of compensation data were achieved. Downstream subscription terminals can subscribe to specific compensation topics according to their own needs, making the compensation process more flexible. At the same time, the centralized processing method also facilitates unified monitoring, scheduling, and maintenance of the data compensation process, improving the manageability and maintainability of the system.

[0039] In one embodiment, this application also provides a data compensation device to address the problem that existing data compensation schemes require each subscriber to perform data verification with the end database separately. Because the data verification strategies, comparison granularity, or processing logic of each subscriber may be inconsistent, this can easily lead to differences in the compensated data among different subscribers, affecting the consistency of downstream data usage. A schematic diagram of the specific structure of this data compensation device is shown below. Figure 3 As shown, it includes: an abnormal data determination unit 31, a compensation data search unit 32, a message push unit 33, and a compensation unit 34.

[0040] The abnormal data determination unit 31 is used to perform data verification between the subscription terminal data and the data in the source database through the first subscription terminal to determine abnormal data. The first subscription terminal is any subscription terminal in the downstream subscription terminal group of the source database. The compensation data lookup unit 32 is used to look up the compensation data corresponding to the abnormal data in the source database based on the abnormal data. The message push unit 33 is used to encapsulate the compensation data according to a preset message format and push it to the corresponding compensation topic of the message middleware. The compensation topic is set according to the information of each data table in the source database and is used to distinguish compensation data from different sources. The compensation unit 34 is used to send a data compensation notification carrying a compensation topic to each subscription terminal in the downstream subscription terminal group, so that each subscription terminal in the downstream subscription terminal group can find the compensation data in the message middleware according to the compensation topic and perform data compensation.

[0041] In one implementation, the abnormal data determination unit 31 is specifically used to: perform a full field comparison between the subscription terminal data and the data in the source database according to a preset verification period, and determine the data with inconsistent field values ​​as abnormal data.

[0042] In one implementation, the compensation data lookup unit 32 is specifically used to: query and obtain complete compensation data from the source database using the primary key of the abnormal data.

[0043] In one implementation, the message push unit 33 is specifically configured to: authorize access permissions for the compensation topic to each subscription terminal in the downstream subscription terminal group; determine the start time node of data compensation; generate a data compensation notification according to the start time node, and send the data compensation notification to each subscription terminal in the downstream subscription terminal group.

[0044] In one implementation, the compensation unit 34 is specifically configured to: determine the original subscription time node of each subscription terminal in the downstream subscription terminal group on the original data topic; when the original subscription time node is less than the start time node, each subscription terminal switches the original data topic to the compensation topic for data consumption; after completing data consumption, the compensation topic switches back to the original data topic.

[0045] In one embodiment, the compensation unit 34 is specifically configured to: when the original subscription time node is greater than the start time node, each subscription terminal switches the original data topic to the compensation topic for data consumption; after data consumption is completed and before switching the compensation topic back to the original data topic, the original subscription time node of the original data topic is reverted to a second time node before the start time node according to a preset duration.

[0046] Using the data compensation device provided in this application embodiment, when data verification and compensation are required, a first subscription terminal can verify the data of the subscription terminal and the data in the source database to identify abnormal data. The first subscription terminal is any subscription terminal in the downstream subscription terminal group of the source database. Based on the abnormal data, compensation data corresponding to the abnormal data is searched in the source database. The compensation data is encapsulated according to a preset message format and pushed to the corresponding compensation topic in the message middleware. The compensation topic is set according to the information of each data table in the source database and is used to distinguish compensation data from different sources. A data compensation notification carrying the compensation topic is sent to each subscription terminal in the downstream subscription terminal group, so that each subscription terminal in the downstream subscription terminal group can search for compensation data in the message middleware according to the compensation topic and perform data compensation. The data compensation method provided in this application has several advantages. First, by having a single subscription terminal (the first subscription terminal) centrally execute data verification and compensation data query operations, the database connections and query requests that originally required all downstream subscription terminals to establish separately are merged into one. This significantly reduces the number of concurrent database connections and query load, effectively avoiding database performance degradation or service congestion caused by multiple terminals accessing the system simultaneously. This improves the stability and reliability of the entire data flow system. Furthermore, by distributing the acquisition of verification and compensation data to a single terminal, other downstream subscription terminals do not need to repeatedly execute the same data comparison and database query logic. This avoids the redundant consumption of computing resources, network bandwidth, and storage space, reducing the overall operating cost of the system and improving the overall efficiency of the data processing link. Second, all downstream subscription terminals obtain compensation data provided by the first subscription terminal from the same compensation topic in the message middleware. This data is uniformly verified and formatted according to standards, fundamentally eliminating data differences caused by inconsistencies in verification rules, processing logic, or data sources among different terminals. This ensures a high degree of consistency in the final data content of all subscription terminals, providing an accurate and reliable data foundation for downstream services. Finally, by creating independent compensation topics for different data tables, isolation and granular management of compensation data were achieved. Downstream subscription terminals can subscribe to specific compensation topics according to their own needs, making the compensation process more flexible. At the same time, the centralized processing method also facilitates unified monitoring, scheduling, and maintenance of the data compensation process, improving the manageability and maintainability of the system.

[0047] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 4At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0048] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0049] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0050] The processor reads the corresponding computer program from non-volatile memory into main memory and then runs it, forming a data compensation device at the logical level. The processor executes the program stored in memory and specifically performs the following operations: The first subscription terminal verifies the data between itself and the source database to identify abnormal data. The first subscription terminal is any one of the downstream subscription terminal groups in the source database. Based on the abnormal data, compensation data corresponding to the abnormal data is searched in the source database. The compensation data is encapsulated according to a preset message format and pushed to the corresponding compensation topic in the message middleware. The compensation topic is set based on the information in each data table of the source database and is used to distinguish compensation data from different sources. A data compensation notification carrying the compensation topic is sent to each subscription terminal in the downstream subscription terminal group, enabling each subscription terminal in the downstream subscription terminal group to search for compensation data in the message middleware based on the compensation topic and perform data compensation.

[0051] The above is as stated in this application. Figure 4The data compensation electronic device method disclosed in the illustrated embodiments can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0052] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0053] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by a portable electronic device including multiple applications, enable the portable electronic device to perform... Figure 1 The method of the illustrated embodiment is specifically used to perform the following operations: The first subscription terminal verifies the data between itself and the source database to identify abnormal data. The first subscription terminal is any one of the downstream subscription terminal groups in the source database. Based on the abnormal data, compensation data corresponding to the abnormal data is searched in the source database. The compensation data is encapsulated according to a preset message format and pushed to the corresponding compensation topic in the message middleware. The compensation topic is set based on the information in each data table of the source database and is used to distinguish compensation data from different sources. A data compensation notification carrying the compensation topic is sent to each subscription terminal in the downstream subscription terminal group, enabling each subscription terminal in the downstream subscription terminal group to search for compensation data in the message middleware based on the compensation topic and perform data compensation.

[0054] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0055] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0056] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0057] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0058] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0059] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0060] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0061] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0062] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0063] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data compensation method, characterized in that, include: The first subscription terminal verifies the data of the subscription terminal against the data in the source database to identify abnormal data. The first subscription terminal is any subscription terminal in the downstream subscription terminal group of the source database. Based on the abnormal data, search the source database for compensation data corresponding to the abnormal data; The compensation data is encapsulated according to a preset message format and pushed to the corresponding compensation topic in the message middleware. The compensation topic is set according to the information of each data table in the source database and is used to distinguish compensation data from different sources. A data compensation notification carrying a compensation topic is sent to each subscription terminal in the downstream subscription terminal group, so that each subscription terminal in the downstream subscription terminal group can find the compensation data in the message middleware according to the compensation topic and perform data compensation.

2. The method according to claim 1, characterized in that, The step of verifying the data on the subscription terminal and the data in the source database through the first subscription terminal to identify abnormal data specifically includes: According to the preset verification cycle, the data from the subscription terminal is compared with the data in the source database, and data with inconsistent field values ​​is identified as abnormal data.

3. The method according to claim 1, characterized in that, The step of searching for compensation data corresponding to the abnormal data in the source database based on the abnormal data specifically includes: Using the primary key of the abnormal data, query and obtain complete compensation data from the source database.

4. The method according to claim 1, characterized in that, The step of sending a data compensation notification carrying a compensation topic to each subscription terminal in the downstream subscription terminal group specifically includes: Authorize each subscription terminal in the downstream subscription terminal group to access the compensation topic; Determine the start time for data compensation; Based on the start time node, a data compensation notification is generated and sent to each subscription terminal in the downstream subscription terminal group.

5. The method according to claim 4, characterized in that, Each subscription terminal in the downstream subscription terminal group searches for compensation data in the message middleware according to the compensation topic and performs data compensation, specifically including: Determine the original subscription time point of each subscription terminal in the downstream subscription terminal group for the original data topic; When the original subscription time node is less than the start time node, each subscription terminal will switch the original data topic to the compensation topic for data consumption. After the data consumption is complete, the compensation topic is switched back to the original data topic.

6. The method according to claim 5, characterized in that, Each subscription terminal in the downstream subscription terminal group searches for compensation data in the message middleware according to the compensation topic and performs data compensation, which also includes: When the original subscription time node is greater than the start time node, each subscription terminal will switch the original data topic to the compensation topic for data consumption. After data consumption is completed and before the compensation topic is switched back to the original data topic, the original subscription time node of the original data topic is reverted to a second time node before the start time node according to a preset duration.

7. A data compensation device, characterized in that, include: An abnormal data determination unit is used to perform data verification between the subscription terminal data and the data in the source database through a first subscription terminal to determine abnormal data, wherein the first subscription terminal is any subscription terminal in the downstream subscription terminal group of the source database; The compensation data lookup unit is used to look up the compensation data corresponding to the abnormal data in the source database based on the abnormal data. The message push unit is used to encapsulate the compensation data according to a preset message format and push it to the corresponding compensation topic of the message middleware. The compensation topic is set according to the information of each data table in the source database and is used to distinguish compensation data from different sources. The compensation unit is used to send a data compensation notification carrying a compensation topic to each subscription terminal in the downstream subscription terminal group, so that each subscription terminal in the downstream subscription terminal group can find the compensation data in the message middleware according to the compensation topic and perform data compensation.

8. The apparatus according to claim 7, characterized in that, The abnormal data determination unit is specifically used for: According to the preset verification cycle, the data from the subscription terminal is compared with the data in the source database, and data with inconsistent field values ​​is identified as abnormal data.

9. A data compensation device, comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1 to 6.

Images