Data restoration method and device based on data comparison and medium

Abstract

The invention provides a data restoration method and device based on data comparison and a medium, and relates to the technical field of data processing. The data packets are distributed to a destination library repair component through message middleware, the repair component starts multi-thread parallel processing, and each thread receives a data packet task from the middleware in sequence; according to the identifier of the data packet, executing interval query by using a primary key, obtaining a corresponding data block in a target library, and forming an ordered data set consistent with a source end structure; performing linear traversal comparison on the same primary key interval data blocks of the source end and the target library to generate a difference data list; and repairing and synchronizing the difference data. According to the method, the data interval is divided through the primary key or the hidden column, and multi-thread parallel processing is combined, so that full-table scanning is avoided, the data volume of single operation is reduced, and the repair efficiency is improved. The multi-thread parallel processing can greatly improve the processing speed, improve the data recovery efficiency and reduce the processing time.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data repair method, device and medium based on data comparison. Background Art

[0002] With the rapid advancement of medical technology and the increasing application of information technology, medical institutions are increasingly relying on digital and information-based tools to significantly improve the quality of medical services and optimize operational efficiency. In a medical data aggregation environment, data, like small streams converging into a vast ocean, continuously grows and changes. With source databases constantly generating massive amounts of medical-related data, existing data processing methods struggle to efficiently and orderly handle this data. Directly operating on large amounts of data without segmentation leads to data chaos, extremely low query and processing efficiency, and an inability to meet real-time or rapid processing requirements.

[0003] In related technologies, data discrepancies may exist between the source and destination databases. These discrepancies, such as data entry errors, loss during transmission, or modification, can lead to inaccurate data in the destination database without effective comparison and repair mechanisms. Table structures vary across databases, with some having primary keys and others not, making unified and efficient processing difficult. Failure to select appropriate block partitioning and processing criteria based on the database's actual structure can render data processing solutions ineffective across multiple databases. Summary of the Invention

[0004] The present invention provides a data repair method based on data comparison. The method divides data into ordered data intervals based on block query of primary keys or hidden columns, and then utilizes multi-threaded parallel processing. Each thread processes a data packet task in sequence, thereby achieving orderly and efficient data processing.

[0005] Methods include: S101: Initiate a block query on the source database based on the primary key or hidden column to generate data intervals sorted by the primary key; S102: Encapsulate the query results into data packets according to data blocks; S103: The data packet is distributed to the target database repair component through the message middleware. The repair component starts multi-threaded parallel processing. Each thread receives a data packet task from the middleware in sequence. S104: Each thread performs interval query in the destination database based on the identifier of the data packet and the primary key to obtain the corresponding data block in the destination database, thereby forming an ordered data set consistent with the source end structure. S105: Perform linear traversal comparison on the data blocks in the same primary key interval of the source and target databases to generate a list of difference data; S106: Perform repair and synchronization operations on the difference data.

[0006] It should be further explained that step S101 specifically includes: selecting a partitioning benchmark according to the database type and table structure; If the table defines a primary key, the primary key is used as the basis for sorting and partitioning; If there is no primary key, a hidden column supported by the database is used as the unique identifier; Get the range of primary key or hidden columns to determine the global boundaries of data partitions.

[0007] It should be further explained that step S101 also includes: formulating a block strategy according to data distribution characteristics; Among them, the interval is divided according to a fixed step size; Or, adjust the interval size according to the data density; Generate a block list, execute block query and extract data, initiate a query for each data interval, extract ordered data blocks, and verify block integrity.

[0008] It should be further explained that in step S102 , a start position, an end position, and an identifier for orderly transmission and comparison of primary key ranges are defined in each data packet header.

[0009] It should be further explained that step S103 specifically includes: calling the message middleware to create a topic for data packet distribution; Publish the encapsulated data packets to the middleware in the order of primary keys, with each data packet as an independent message; Start multiple threads and define each thread to execute the preset operation process; Each thread pulls tasks from the middleware in sequence. The processes executed after the thread pulls the task include but are not limited to: data packet parsing, destination library query, difference comparison and repair, and task failure, triggering the retry mechanism or recording the error log.

[0010] It should be further explained that step S104 specifically includes: parsing the data packet identification information and constructing a primary key interval query statement; Execute the query and obtain the target database data block; Arrange the query results in ascending order by primary key, and keep them consistent with the order of the source data blocks; Convert the query results into a serialized format consistent with the source data block and cache them.

[0011] It should be further explained that S105 specifically includes: comparing field contents line by line and recording inconsistent entries; Among them, missing or redundant data are marked and a list of difference data is generated.

[0012] It should be further explained that the repair operation in S106 includes: overwriting inconsistent fields in the target database, supplementing records that exist on the source side but are missing in the target database, and removing redundant data in the target database.

[0013] According to another embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data repair method based on data comparison when executing the program.

[0014] According to another embodiment of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data repair method based on data comparison are implemented.

[0015] It can be seen from the above technical solutions that the present invention has the following advantages: The data comparison-based data repair method provided in this application can quickly process massive amounts of data through block query and multi-threaded parallel processing. In the medical business system scenario, faced with the large amount of patient diagnosis and treatment data generated every day, block query can obtain data in an orderly manner, and multi-threaded parallel processing can significantly increase processing speed, improve data repair efficiency, and reduce processing time.

[0016] By performing a linear traversal comparison of the source and destination database data, generating a list of difference data, and executing a repair and synchronization operation, we ensure consistency between the destination and source database data, guaranteeing data accuracy. We also select a partitioning basis based on the database type and table structure, utilizing either the primary key or, when no primary key exists, hidden columns. We also develop a partitioning strategy based on data distribution characteristics. We utilize message-based middleware for data packet distribution, configuring topics, partitions, and other strategies to ensure sequential data transmission. We also record message offsets to ensure continued processing after interruptions and recovery, ensuring data processing reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for the description. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 It is a flow chart of a data repair method based on data comparison; Figure 2 Schematic diagram of an electronic device. DETAILED DESCRIPTION

[0019] The data comparison-based data repair method provided in this application is based on the scenario of data collection and aggregation. The source database of the medical business system continuously generates massive amounts of medical-related data every day. Based on the data comparison and repair method, data reconciliation is performed, and the data in the target database is differentiated to check for omissions and fill gaps. Missing or updated data is updated to the data lake, ensuring consistency between the data lake and the source database, and improving the data operation and maintenance efficiency of the medical system.

[0020] The specific steps of the data repair method based on data comparison will be described in detail below. For the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are provided to facilitate a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details.

[0021] It should be understood that when used in this specification, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their collections. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0022] The phrases "one embodiment" or "some embodiments" described in this application mean that the specific features, structures, or characteristics described in the embodiment are included in one or more embodiments of the application. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in other embodiments," etc. that appear in different places in this application do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized.

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0024] See also Figure 1 FIG2 is a flow chart of a data repair method based on data comparison in a specific embodiment, the method comprising: S101: Initiate a block query on the source database based on the primary key or hidden column to generate data intervals arranged in order by the primary key.

[0025] In some embodiments, the source database uses a partitioning rule based on the primary key or hidden column used to uniquely identify records in the database table. For example, partitioning can be performed based on the range of primary key values, with each range of primary key values ​​corresponding to a data interval. A query operation is then executed to partition the source database data into multiple data intervals, ordered by primary key, according to the defined rule. This allows for orderly data retrieval and facilitates subsequent processing.

[0026] In this embodiment, step S101 specifically includes selecting a partitioning basis based on the database type and table structure. If a primary key is defined for the table, the primary key is used as the basis for sorting and partitioning. If no primary key is defined, a hidden column supported by the database is used as a unique identifier to ensure global uniqueness of the data partitioning.

[0027] In this embodiment, the global boundaries of data partitions can be determined by the following operations when obtaining the range of a primary key or hidden column. A query statement is executed to obtain the minimum and maximum values ​​of the primary key. Of course, if a hidden column is used, the minimum and maximum identifier values ​​of the physical storage can be obtained using a built-in database function or system view.

[0028] Finally, a partitioning strategy is developed based on the data distribution characteristics. Specifically, this can be done by dividing the intervals into fixed steps. The interval size can also be adjusted based on data density. For example, by sampling and analyzing the distribution of primary keys, the partitioning range can be reduced in high-density areas to avoid excessively large individual blocks. Finally, a partition list is generated to ensure that adjacent blocks cover all data and do not overlap.

[0029] This embodiment performs a block query and extracts data. It can initiate a query for each data interval to extract ordered data blocks. If hidden columns are used, block partitioning is implemented using database-specific syntax. To optimize performance, indexes are used to accelerate queries and limit the number of rows returned by a single query.

[0030] S102: Encapsulate the query results into data packets according to the data blocks.

[0031] In this embodiment, a starting position and an ending position may be defined in each data packet header, as well as an identifier for orderly transmission and comparison of primary key ranges.

[0032] S103: The data packet is distributed to the target database repair component through the message middleware. The repair component starts multi-threaded parallel processing. Each thread receives a data packet task from the middleware in sequence.

[0033] In some specific embodiments, the message middleware is first configured and initialized. Here, the appropriate message middleware can be selected and parameters configured. Specifically, the parameters can be topics, and dedicated topics can be created for data packet distribution.

[0034] Optionally, if using Kafka, partition the topic by primary key range or data block number to ensure that tasks for the same data block are assigned to the same partition to ensure orderliness; In this embodiment, the encapsulated data packets are also published to the middleware in primary key order. In other words, each data packet is treated as a separate message, with the message body containing the data block content and the packet header identifier. Metadata can also be attached to facilitate destination routing and log tracking.

[0035] The target library repair in this embodiment can start multiple threads, and each thread can be connected to a specified topic or queue.

[0036] Optionally, if a partitioning strategy is used, each thread is bound to a separate partition, ensuring that messages in the same partition are processed sequentially by the same thread.

[0037] This embodiment also allows for sequential data packet tasks. Specifically, each thread pulls tasks from the middleware in order. Tasks can be processed by single-threaded consumption within a partition or by using the first-in-first-out (FIFO) feature of a queue, ensuring that data blocks are processed in primary key order.

[0038] This embodiment also records the offset of the processed message, and continues consumption from the breakpoint after recovery to avoid duplication or omission.

[0039] After a thread receives a task, it first extracts the packet header identifier and the data within the block. It then queries the corresponding data block in the target database, performs a linear comparison, and generates a difference operation instruction. It then sends an ACK (acknowledgement signal) to the middleware, marking the message as successfully processed. If the task fails, a retry mechanism is triggered or an error log is recorded.

[0040] As can be seen, the message-based middleware is used to send the encapsulated data packets to the target database repair component. The target database repair component then activates a multi-threaded mechanism, with multiple threads running simultaneously. Each thread sequentially receives a data packet task from the message-based middleware. This parallel processing method improves data processing efficiency and accelerates the repair process.

[0041] S104: Each thread performs interval query in the destination database based on the identifier of the data packet and the primary key to obtain the corresponding data block in the destination database, thereby forming an ordered data set consistent with the source end structure.

[0042] This embodiment extracts header metadata from the received data packet by parsing the data packet identification information, including: a start position identifier, an end position identifier, and table structure metadata, and constructs a primary key interval query statement.

[0043] Of course, if you use hidden columns, you can replace them with physical address query syntax supported by the database.

[0044] This embodiment also executes a query and retrieves the target database data block. The thread here initiates a query request to the target database. Specifically, indexing can be used, and ensuring that the primary key index is effective accelerates the query. If the amount of data is too large, paging queries are used to avoid memory overflows. This embodiment also sorts the query results in ascending order by primary key to ensure consistency with the source data block order. Checks for a match in the number of columns, column names, and data types; if the structures are inconsistent, an exception is triggered, an error log is recorded, and the current task is paused.

[0045] After the query, an ordered dataset is generated and cached. As can be seen, each thread that receives a task uses the primary key in the destination database to perform a range query based on the identifier carried in the data packet. Based on the primary key range identified in the data packet, the corresponding data block is searched in the destination database, thereby obtaining an ordered dataset that is consistent with the source database structure, ensuring consistency in structure and order between the source and destination databases.

[0046] S105: Perform linear traversal comparison on the data blocks in the same primary key interval of the source and target databases to generate a list of difference data.

[0047] In this embodiment, a linear traversal comparison is performed on the data blocks within the same primary key interval in the source and destination databases. Specifically, the field contents can be compared row by row, and inconsistent entries are recorded. Missing or redundant data can also be marked, and a list of difference data is generated.

[0048] S106: Perform repair and synchronization operations on the difference data.

[0049] This embodiment operates by performing repair operations based on the difference list. Specifically, this can involve overwriting inconsistent fields in the destination database, inserting records that exist on the source but are missing from the destination database, or deleting redundant data in the destination database. Upon completion, the next data block processing flow is triggered until all data blocks are synchronized.

[0050] In this way, based on the generated difference data list, the difference data in the destination database is repaired and synchronized. The correct data on the source side is updated to the corresponding location in the destination database, making the data in the destination database consistent with the source side, and completing the data repair work.

[0051] On the basis of the above embodiment, in order to further improve the reliability of the data repair method based on data comparison provided by the above embodiment, the following is a specific implementable method: This method first queries the source database according to the primary key or hidden column, and then sends the sub-packages to the target database repair component. When encapsulating the data packet, it adds the start position and end position identifiers to the data packet header. This forms ordered data blocks identified by the primary key. The specific method also includes the following steps: Step 501: Query data from the source database based on the primary key or hidden column to obtain the data set to be repaired.

[0052] Step 502: Split the query results into multiple data packets according to preset rules and send them to the target database repair component in batches.

[0053] Step 503: When encapsulating a data packet, identifiers of the start position and the end position are added to the packet header to mark the data range.

[0054] Step 504: Arrange the data in order of the primary key to generate ordered data blocks with the primary key as the unique identifier, so that the data has logical coherence.

[0055] Step 505: Start the core data packet splicing program flow and begin scheduling and executing the data repair task.

[0056] Step 506: Initialize the SQL statement template for data query and define basic query logic.

[0057] Step 507: Dynamically splice SQL statements according to the comparison conditions to achieve accurate data positioning.

[0058] Step 508: Process the result set returned by the query, serialize it and fill it into the corresponding data packet.

[0059] Step 509: Complete the final encapsulation and sending of the data packet, and trigger the receiving and storage operation of the repair component at the destination end.

[0060] The present application also involves extracting data packets from the message middle key based on multiple threads, comparing and processing their own data packets in each thread of the repair component module, using the primary key to query the corresponding data interval, and then performing a linear difference comparison on the two ordered data blocks, kicking out the difference data, and updating the difference data of the destination library.

[0061] The specific method involves the following steps: Step 601: Multiple threads obtain data packets from the message middleware in parallel, and each thread independently takes a data packet task to ensure that there is no conflict in task allocation.

[0062] Step 602: Each thread parses the primary key range identifier in the data packet, and may identify the source and destination data intervals that need to be processed based on the start primary key and the end primary key.

[0063] Step 603: The thread performs interval query in the target database based on the primary key range, extracts the ordered data set corresponding to the source data block, and ensures data structure consistency.

[0064] Step 604: The thread loads the data blocks of the same primary key interval of the source and target databases into the memory, and arranges them row by row in the order of the primary keys to form a basis for linear comparison.

[0065] Step 605: Perform a field-level row-by-row comparison of the source and destination data blocks, mark the difference data, and generate a difference list.

[0066] Step 606: Perform repair operations based on the difference list to delete redundant data in the target database or update inconsistent fields to ensure consistency with the source data.

[0067] Step 607: After the repair is completed, the last repair timestamp of the destination is updated as metadata and a log is recorded for tracking and auditing.

[0068] It can be seen that this embodiment locates data through primary keys, avoids full table scans, and reduces invalid comparisons. Based on the packet sending mechanism, the single transmission load is reduced and network utilization is improved. Using multi-threading to receive tasks from the message middleware in parallel, the efficiency of large-scale data repair is significantly improved. Through primary key interval query and linear comparison, full table scans are avoided and field-level differences are quickly located. This embodiment can delete or update the difference data on the destination end. When repairing data conflicts or loss problems between the source and target libraries, the accuracy of business logic is ensured. Through metadata updates and logging, the verifiability and rollback capabilities of the repair process are supported.

[0069] The above method of this embodiment can be applied to a specific medical business system. For example, in a large comprehensive hospital, the source database of its medical business system generates a massive amount of patient diagnosis and treatment data every day, including basic patient information, diagnosis records, test reports, medication records, etc. The destination database is used for data backup and data analysis.

[0070] This example allows you to initiate a block query on the source database based on the primary key or hidden column, generating data intervals sorted by primary key. For example, in a hospital's patient diagnosis and treatment information table, a block query is performed based on patient ID. Given the large volume of data, a data interval is set for every 1000 patient records. For example, records with patient IDs 1-1000 are the first data interval, 1001-2000 is the second data interval, and so on. By writing a SQL query statement, the corresponding data records are retrieved from the source database according to the specified patient ID range, generating data intervals sorted by patient ID.

[0071] Each patient's medical record within a data interval is considered a data block. Detailed identification information is added to each data block, such as the starting patient ID, ending patient ID, and data block creation time. These data blocks are then encapsulated according to a specific data format to form a data packet.

[0072] In this example, 10 threads can be started, each of which sequentially receives a data packet task from the Kafka topic. Multi-threaded parallel processing can greatly improve data processing efficiency.

[0073] The thread that receives the data packet containing patient IDs 1-1000 searches the patient diagnosis and treatment information table in the destination database for records with patient IDs between 1 and 1000. This method obtains an ordered dataset that matches the structure of the source database, ensuring consistency in structure and order between the source and destination databases.

[0074] This example can start with the first record in the data block and compare each field of each patient record in turn. For example, comparing the patient's diagnosis record field reveals that the diagnosis of patient ID 10 on the source side is a cold, while the corresponding record in the destination database shows a viral cold. Comparing the medication record field reveals that the source side records the patient's use of a certain antibiotic, while the destination database does not. These discrepancies in the records and field information are recorded, and a list of discrepancies is generated, clearly outlining the data that needs to be repaired.

[0075] This technique allows you to repair and synchronize the discrepancies in the destination database based on the generated differential data list. For the record with patient ID 10 in the previous example, the correct diagnosis (cold) and medication record from the source are updated to the corresponding record in the destination database. By writing appropriate database update statements, each discrepancy in the differential data list is processed one by one, ensuring consistency between the destination database and the source. This completes the repair of massive amounts of medical data, ensuring the accuracy and consistency of data in medical business systems and providing reliable data support for subsequent medical data analysis and decision-making.

[0076] like Figure 2 As shown, the present application also provides an electronic device, including a display module 103, a memory 102, a processor 101, and a computer program stored in the memory and executable on the processor 101, wherein the processor 101 implements the steps of a data repair method based on data comparison when executing the program.

[0077] In the embodiments of the present invention, electronic devices include, but are not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or claimed herein.

[0078] In the embodiment of the present application, the processor 101 can be implemented by using at least one of an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), a processor, a controller, a microcontroller, a microprocessor, and an electronic unit designed to perform the functions described herein. In some cases, such an embodiment can be implemented in a controller. For software implementation, an embodiment such as a process or function can be implemented with a separate software module that allows the execution of at least one function or operation. The software code can be implemented by a software application (or program) written in any appropriate programming language, and the software code can be stored in a memory and executed by a controller.

[0079] The display module 103 is used to display information input by the user or information provided to the user. The display module 103 may include a display panel, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc.

[0080] The memory 102 can be used to store software programs and various data. The memory 102 can include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0081] The present application also provides a storage medium having a computer program stored thereon, which implements the steps of the data repair method based on data comparison when the computer program is executed by a processor.

[0082] The storage medium can be any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0083] In the context of storage media, a readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0084] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data repair method based on data comparison, characterized in that: Methods include: S101: Initiate a block query on the source database based on the primary key or hidden column to generate data intervals sorted by the primary key; S102: Encapsulate the query results into data packets according to data blocks; S103: The data packet is distributed to the target database repair component through the message middleware. The repair component starts multi-threaded parallel processing. Each thread receives a data packet task from the middleware in sequence. S104: Each thread performs interval query in the destination database based on the identifier of the data packet and the primary key to obtain the corresponding data block in the destination database, thereby forming an ordered data set consistent with the source end structure. S105: Perform linear traversal comparison on the data blocks in the same primary key interval of the source and target databases to generate a list of difference data; S106: Perform repair and synchronization operations on the difference data.

2. The data repair method based on data comparison according to claim 1, characterized in that: Step S101 specifically includes: selecting a partitioning benchmark according to the database type and table structure; If the table defines a primary key, the primary key is used as the basis for sorting and partitioning; If there is no primary key, a hidden column supported by the database is used as the unique identifier; Get the range of primary key or hidden columns to determine the global boundaries of data partitions.

3. The data repair method based on data comparison according to claim 2, characterized in that: Step S101 also includes: formulating a block partitioning strategy based on data distribution characteristics; Among them, the interval is divided according to a fixed step size; Or, adjust the interval size according to the data density; Generate a block list, execute block query and extract data, initiate a query for each data interval, extract ordered data blocks, and verify block integrity.

4. The data repair method based on data comparison according to claim 1, characterized in that: In step S102 , a starting position, an ending position, and an identifier for orderly transmission and comparison of primary key ranges are defined in each data packet header.

5. The data repair method based on data comparison according to claim 1, characterized in that: Step S103 specifically includes: calling the message middleware to create a topic for data packet distribution; Publish the encapsulated data packets to the middleware in the order of primary keys, with each data packet as an independent message; Start multiple threads and define each thread to execute the preset operation process; Each thread pulls tasks from the middleware in sequence. The processes executed after the thread pulls the task include but are not limited to: data packet parsing, destination library query, difference comparison and repair, and task failure, triggering the retry mechanism or recording the error log.

6. The data repair method based on data comparison according to claim 1, characterized in that: Step S104 specifically includes: parsing the data packet identification information and constructing a primary key interval query statement; Execute the query and obtain the target database data block; Arrange the query results in ascending order by primary key, and keep them consistent with the order of the source data blocks; Convert the query results into a serialized format consistent with the source data block and cache them.

7. The data repair method based on data comparison according to claim 1, characterized in that: S105 specifically includes: comparing field contents line by line and recording inconsistent entries; Among them, missing or redundant data are marked and a list of difference data is generated.

8. The data repair method based on data comparison according to claim 1, characterized in that: The repair operations in S106 include: Overwrite inconsistent fields in the destination database, supplement records that exist on the source side but are missing in the destination database, and remove redundant data in the destination database.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the data repair method based on data comparison as described in any one of claims 1 to 8 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data repair method based on data comparison as claimed in any one of claims 1 to 8 are implemented.

Images