Database multi-source heterogeneous data synchronization method and device

Abstract

The embodiment of the invention discloses a database multi-source heterogeneous data synchronization method and device.The method comprises the steps that a data change event is monitored through an RPA robot deployed in a server where a source database is located, and when change operation of structured data or unstructured data is detected, an initial data set is generated; performing fragmentation processing on the initial data set to generate a plurality of data fragmentation tasks, and allocating the data fragmentation tasks to distributed computing nodes to execute parallel format conversion operation so as to generate a target data set; detecting a task execution state of the distributed computing node, and when a generation completion event of a target data set is detected, activating a data synchronization channel and transmitting the target data set to a target database; and performing conflict detection and identification based on an RPA conflict processing strategy configured in the target database, when a primary key conflict or a uniqueness constraint conflict is identified, updating the target database based on preset conflict merging indication information, and generating a data synchronization log.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and specifically to a method and device for synchronizing multi-source heterogeneous data in a database. Background Art

[0002] With the rapid development of information technology, RPA database technology is playing an increasingly important role in data processing and management. RPA can simulate human operations, automate processes, and improve work efficiency and accuracy. In database management, it also provides new solutions for tasks such as data migration and synchronization.

[0003] In the current context of RPA database technology, existing technologies still have the following flaws: Monitoring of data changes in the source database is not timely and comprehensive enough, and it is unable to simultaneously account for changes in structured and unstructured data, resulting in untimely updates of some data and affecting data integrity; during the data processing process, data format conversion efficiency is low, and there is a lack of parallel processing capabilities, which cannot meet the needs of rapid large-scale data processing; and when data is synchronized to the target database, there is a lack of effective detection and processing mechanisms for potential conflicts, which can easily lead to data inconsistencies and affect the normal operation of business systems. Summary of the Invention

[0004] The embodiments of the present application provide a method and device for synchronizing multi-source heterogeneous data in a database, which is used to effectively improve the timeliness, efficiency and consistency of data processing through real-time monitoring of data changes by RPA robots, distributed parallel processing and precise conflict detection and processing mechanisms.

[0005] In a first aspect, an embodiment of the present application provides a database multi-source heterogeneous data synchronization method, which is applied to a database multi-source heterogeneous data synchronization device, and the method includes: monitoring data change events by an RPA robot deployed on the server where the source database is located, and generating an initial data set when a change operation of structured data or unstructured data is detected; sharding the initial data set to generate multiple data sharding tasks, and assigning the data sharding tasks to distributed computing nodes to perform parallel format conversion operations to generate a target data set; detecting the task execution status of the distributed computing nodes, and when a generation completion event of the target data set is detected, activating a data synchronization channel and transmitting the target data set to a target database; performing conflict detection and identification based on the RPA conflict handling strategy configured in the target database, and when a primary key conflict or a uniqueness constraint conflict is identified, updating the target database based on preset conflict merge indication information, and generating a data synchronization log.

[0006] In a second aspect, an embodiment of the present application provides a database multi-source heterogeneous data synchronization device, comprising: processor; a storage device having a computer program stored thereon, When the computer program is executed by the processor, the processor implements any of the methods for synchronizing multi-source heterogeneous data in a database.

[0007] An embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method for synchronizing multi-source heterogeneous data in a database are implemented.

[0008] It can be seen that the embodiments of the present application have the following beneficial effects: by monitoring data changes through the RPA robot deployed on the source database server, it is possible to capture changes in structured and unstructured data in real time, generate the initial data set in a timely manner, and ensure the timeliness and integrity of the data. The initial data set is fragmented and distributed to the distributed computing nodes to perform parallel format conversion, which greatly improves the data processing efficiency, fully utilizes the advantages of distributed computing, and quickly generates the target data set. By detecting the execution status of the distributed computing node tasks in real time, the data synchronization channel is activated once the target data set is generated, and the data can be accurately and efficiently transmitted to the target database to ensure the timeliness of data flow. Conflict detection and identification are performed in the target database based on the configured RPA conflict handling strategy, and the database is updated and synchronization logs are generated according to the preset information, which can effectively solve the conflict problem in data synchronization, ensure the consistency and accuracy of the data, improve the reliability and stability of the entire data synchronization process, and provide solid data support for the stable operation of the RPA business system. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 This is a flowchart of a method for synchronizing multi-source heterogeneous data in a database provided in an embodiment of the present application.

[0010] Figure 2 A schematic diagram of the basic structure of a database multi-source heterogeneous data synchronization device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0011] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0012] See also Figure 1 As shown, this figure is a flowchart of a database multi-source heterogeneous data synchronization method provided by an embodiment of the present application, which can be applied to a database multi-source heterogeneous data synchronization device. Figure 1 As shown, the method may include steps 110 to 140.

[0013] Step 110: The RPA robot deployed on the server where the source database is located monitors data change events. When a change operation on structured data or unstructured data is detected, an initial data set is generated.

[0014] In an embodiment of the present application, taking the database system of a large e-commerce enterprise as an example, the source database of the database system stores various types of product information, order information and other data. The RPA robot deployed on the server where the source database is located is always in a listening state. For example, when a new product is entered into the system, which is an insert operation of structured data; or the price of a certain product is modified, which is an update operation of structured data; or a deleted order record, which is a delete operation of unstructured data, etc., the RPA robot will take effect and generate an initial data set. The above initial data set contains the specific data content that has changed in the database, providing original materials for subsequent data synchronization work.

[0015] As an optional embodiment, step 110 includes: Step 111: The RPA robot performs real-time parsing on the operation log of the source database to extract a set of event records containing data insertion, update, or deletion operations.

[0016] In the above application scenario, the source database logs every operation. The RPA robot reads these operation logs in real time and parses them in an orderly and comprehensive manner. For example, the operation log records information such as the time, operator, operation type, and the data table involved. The RPA robot carefully selects records where the operation type is insert, update, or delete. For example, if the operation log shows that user A inserted a new product record into the product information table at a certain moment, the RPA robot will extract this record and place it in the event record collection. In this way, all relevant operation records are collected to form a complete event record collection, providing an accurate data source for subsequent processing.

[0017] Step 112: performing a comparison analysis based on the transaction timestamps in the event record set and the previous synchronization time of the target database, filtering out the unsynchronized transaction operation records from the event record set based on the comparison analysis result, and generating a target transaction operation set.

[0018] In the aforementioned database system, the target database has previously undergone a synchronization operation, and the completion time of that operation is recorded as the previous synchronization time. The RPA robot extracts the transaction timestamp of each record in the event record set and compares it with the previous synchronization time. For example, if the event record set contains a product update record with a transaction timestamp that shows a time after the most recent synchronization, this record is considered an unsynchronized transaction operation record. By comparing each unsynchronized record one by one, all such unsynchronized records are filtered out to form the target transaction operation set. All records in this set are subject to subsequent synchronization to ensure that no new changes are missed.

[0019] Step 113: The RPA robot simulates a database query operation, generates a data extraction script according to the operation type and operation parameters in the target transaction operation set, and executes the data extraction script to pull the initial data set from the source database.

[0020] For the aforementioned database system, the target transaction operation set contains various types of operation records. The RPA robot generates a corresponding data extraction script based on the operation type and parameters of these records. For example, for a product information insertion operation, the operation parameters may include the product ID, name, and price. Based on this information, the RPA robot generates a data extraction script to extract the complete product information from the source database. The RPA robot then executes this script (which can be thought of as manually executing a query statement in the database) to accurately extract the initial data set related to these operations from the source database. This initial data set contains detailed product information, such as product description and inventory quantity, providing a comprehensive data foundation for subsequent data processing.

[0021] Step 114: Add transaction identifier and timestamp metadata to the initial data set, and cache the initial data set to a distributed storage node.

[0022] In the above application scenario, to better track and manage the data synchronization process, the RPA robot adds transaction identifiers and timestamp metadata to each data record in the initial data set. The transaction identifier uniquely identifies a data change operation, while the timestamp metadata records the specific time the operation occurred. For example, for a product update record, the transaction identifier "TX001" and the timestamp "2023-10-05 10:30:00" are added. The initial data set with these metadata is then cached in distributed storage nodes. Distributed storage nodes can be viewed as data warehouses. Multiple storage nodes can jointly store and manage the above data, ensuring data security and accessibility, and facilitating subsequent operations such as data sharding.

[0023] Step 120: Slice the initial data set to generate multiple data slicing tasks, and distribute the data slicing tasks to distributed computing nodes to perform parallel format conversion operations to generate a target data set.

[0024] Given the massive data volumes of the aforementioned database systems, sharding the initial data set is essential. In an embodiment of the present application, the initial data set is divided into multiple parts according to specific rules, forming multiple data sharding tasks. For example, product information data is sharded by product category, with electronic product data divided into one shard and clothing data into another. These data sharding tasks are then distributed to distributed computing nodes, with each node simultaneously performing format conversion operations to improve processing efficiency and ultimately generate a target data set. This target data set, after format conversion, meets the storage requirements of the target database.

[0025] As another optional embodiment, step 120 includes: Step 121: Based on the data distribution characteristics of the initial data set, the initial data set is divided into multiple initial shards using preset sharding rules, where the sharding rules include data table association, data volume balance, and field dependency.

[0026] In the aforementioned database system, various relationships exist between different data tables, with varying amounts of data, and dependencies between fields. For example, a product information table and an order table are related, meaning that a single product may correspond to multiple orders. The pre-set sharding rules take these factors into account. Regarding table relationships, closely related data is grouped together into the same initial shard. For example, all product information for a particular brand and its related order information are grouped together. Regarding data balance, efforts are made to ensure that the data volumes of each initial shard are roughly equal, preventing excessive processing time due to excessive data volume in any one shard. Regarding field dependencies, for example, a product's price field may depend on its category field, and data for related fields is grouped together into the same shard. Using these rules, the initial data set is rationally divided into multiple initial shards.

[0027] Step 122: Create an independent data conversion task for each initial shard, and distribute the data conversion task to an idle computing node based on the load status of the distributed computing node.

[0028] After generating multiple initial shards, create an independent data conversion task for each shard. For example, for the initial shard of the electronic product category, create a data conversion task for converting the data format of the shard. The system then monitors the load status of the distributed computing nodes in real time. The load status can be measured by indicators such as the node's CPU usage and memory usage. If the CPU usage of a computing node is low and there is a lot of free memory, it means that the node is idle. The system will assign data conversion tasks to these idle computing nodes to ensure that the tasks can be executed efficiently and avoid the situation where a node is overloaded while other nodes are idle.

[0029] Step 123: In each distributed computing node, a preconfigured format conversion rule set is called to perform structured data schema alignment and unstructured data semantic standardization on the allocated initial shards to generate intermediate standardized shards.

[0030] On the distributed computing nodes, the initial shards are processed according to the preconfigured format conversion rule set. For structured data, such as data in the product information table, a schema alignment operation is performed. For example, the data type of the product price field in the source database can be a string, while the target database requires a numeric type. The format conversion rule set will guide the node to convert the data type of the field to conform to the schema requirements of the target database. For unstructured data, such as text information in the product description field, semantic standardization is performed. The key information in the text is identified through a pre-trained model and converted into a unified format. For example, product material information expressed in different ways is uniformly converted into standard terms, and finally an intermediate standardized shard is generated.

[0031] In a preferred embodiment, step 123 includes: Step 1231: Perform field mapping matching on the structured data table in the initial shard, automatically compare the source field name with the field definition of the target database through the RPA script, and generate a field mapping relationship table.

[0032] In the initial sharding of the aforementioned database system, the structured data tables contain various fields. The RPA script automatically compares the source field names with the field definitions in the target database. For example, the "product_name" field in the product information table in the source database might correspond to "product name" in the target database. The RPA script identifies this correspondence and generates a field mapping table that records the one-to-one correspondence between the source and target fields, providing accurate guidance for subsequent data conversion.

[0033] Step 1232: Performing automatic type conversion including string encoding conversion, date format unification, and numerical precision adjustment on fields with inconsistent data types according to the field mapping relationship table to obtain a converted structured data table.

[0034] Based on the generated field mapping table, if a field's data type is inconsistent, automatic type conversion will be performed. For example, if the source database's product launch date field is formatted as "YYYY / MM / DD," but the target database requires "YYYY-MM-DD," the date format will be unified. If the precision of a numeric field differs between the source and target databases, the precision will also be adjusted. These operations result in a converted structured data table that meets the requirements of the target database.

[0035] Step 1233: Perform semantic parsing on the unstructured data blocks in the initial shards, extract key entities and contextual relationships through the pre-trained NLP model, and convert the unstructured data into target structured data blocks containing entity labels and relationship graphs.

[0036] Pre-trained NLP models come into play for unstructured data blocks in the initial shards, such as product description text. The model analyzes the textual content, extracting key entities such as the product's brand and model, as well as the contextual relationships between these entities. For example, if a product description mentions "a certain brand of mobile phone equipped with a certain model of processor," the model will identify entities such as "brand," "mobile phone," and "processor," and construct a relationship graph between them. This information is then organized into target structured data blocks containing entity labels and relationship graphs, facilitating subsequent data processing and storage.

[0037] Step 1234: reorganize the converted structured data table and the target structured data block according to the storage rules of the target database and add a shard identifier and a check code to generate an intermediate standardized shard.

[0038] The processed, converted structured data table and the target structured data block are reassembled according to the target database's storage rules. For example, if the target database requires data to be stored in a specified order, the relevant data will be arranged accordingly. Furthermore, to ensure data accuracy and traceability, a shard identifier and checksum are added to the reassembled data. The shard identifier uniquely identifies the initial shard from which the intermediate standardized shard originated, and the checksum is used to detect errors during data transmission or storage. Finally, the intermediate standardized shard is generated.

[0039] Step 124: Aggregate all intermediate standardized shards to obtain shard aggregation results, perform data integrity verification and deduplication processing on the shard aggregation results, and generate the target data set.

[0040] After each distributed computing node generates intermediate standardized shards, these shards are aggregated, combining all intermediate standardized shards into a single entity to produce the aggregated shard result. This aggregated result is then checked for data integrity, checking for missing or corrupted data. For example, product information is checked for completeness and whether any key product information is missing. Deduplication is also performed to remove duplicate data records. For example, if two identical product records exist, only one is retained. These operations ultimately generate the target data set, which has been rigorously processed to meet the requirements of the target database.

[0041] Step 130: Detecting the task execution status of the distributed computing node, and when detecting a generation completion event of the target data set, activating a data synchronization channel and transmitting the target data set to a target database.

[0042] In the aforementioned database system, real-time monitoring of the task execution status of distributed computing nodes is required. The system continuously checks the completion of each node's data conversion task. When all nodes have completed their data conversion tasks and generated the target dataset, a target dataset generation completion event is triggered. At this point, the system activates the data synchronization channel, which is responsible for securely and accurately transferring the target dataset to the target database, ensuring timely data updates.

[0043] As another optional embodiment, step 130 includes: Step 131: Obtain the execution status parameters of each data shard task in the distributed computing node in real time through the RPA task status collector. The execution status parameters include at least the task progress percentage and the abnormal error code.

[0044] In the distributed computing environment of the aforementioned database system, the RPA task status collector operates continuously, communicating with each distributed computing node to obtain real-time execution status parameters for each data shard task. For example, for a data shard task, the collector will obtain the task progress percentage, such as 70% completion, and the exception error code. If there are no errors, the error code will be blank. Using these parameters, the system can fully understand the execution status of each task and promptly identify potential problems.

[0045] Step 132: When it is identified that the progress percentages of all data sharding tasks have reached the preset percentage and the abnormal error code is empty, a target data set generation completion event is generated, and a data synchronization channel running instruction is activated.

[0046] In this embodiment of the present application, the system sets a preset percentage, such as 95%. When the RPA task status collector reports that the progress percentage of all data sharding tasks has reached this preset percentage and all task exception error codes are empty, it means that all data conversion tasks have been successfully completed and the target data set has been generated. At this point, the system generates a target data set generation completion event and activates the data synchronization channel run instruction, notifying the relevant components to prepare to open the data synchronization channel for data transmission.

[0047] Step 133: According to the data synchronization channel running instruction, a pre-configured channel connector is called to establish a bidirectional encrypted channel with the target database. The channel connector loads a corresponding communication protocol driver based on the type of the target database.

[0048] As you can understand, upon receiving the data synchronization channel execution instruction, the system invokes the preconfigured channel connector. Different target database types may require different communication protocol drivers for connection. For example, if the target database is a relational database, the channel connector will load the appropriate communication protocol driver for that database. This communication protocol driver establishes a bidirectional encrypted channel with the target database, ensuring data security and integrity during data transmission and preventing data theft or tampering.

[0049] Step 134: Arrange the target data set in the distributed storage node into a transmission sequence according to the order of the shard identifiers, and inject a flow control strategy for optimizing network congestion.

[0050] In the aforementioned database system, distributed storage nodes store the target data set. To ensure orderly and efficient data transmission to the target database, the target data set is sequenced according to the order of its shard identifiers. For example, data shards for different product categories are arranged in a specific order. Furthermore, to prevent network congestion from impacting data transmission speed and quality, flow control strategies are implemented. For example, the transmission rate of each data shard is dynamically adjusted based on the network's real-time bandwidth to ensure optimal utilization of network resources.

[0051] In another preferred technical solution, step 134 includes: Step 1341: Parse the shard identifier of each data shard in the target data set, extract the global sequence number field and the shard generation timestamp, generate the initial transmission sequence in ascending order of the global sequence number field, and calculate the waiting transmission time of each data shard based on the difference between the shard generation timestamp and the current system timestamp.

[0052] In the target data set, each data shard has its own shard identifier. The system parses these identifiers to extract the global sequence number field and the shard generation timestamp. For example, a shard identifier might contain "003_2023-10-05 11:00:00," where "003" is the global sequence number field and "2023-10-05 11:00:00" is the shard generation timestamp. The initial transmission sequence is generated by sorting the global sequence number fields in ascending order. The shard generation timestamp is then subtracted from the current system timestamp to calculate the waiting time for each data shard to be transmitted, providing a basis for subsequent priority adjustments.

[0053] Step 1342: When the waiting transmission time exceeds the preset transmission time threshold, the priority weight of the corresponding data segment is increased to the first preset level, and the real-time network parameter set between the distributed storage node and the target database is synchronously collected. The network parameter set includes a normalized bandwidth utilization coefficient, delay attenuation coefficient and packet loss penalty coefficient.

[0054] If the waiting time for a data shard exceeds the preset transmission time threshold (for example, if the preset threshold is 10 minutes and the waiting time for a data shard reaches 15 minutes), the priority weight of the data shard will be increased to the first preset level. At the same time, the system collects a set of real-time network parameters between the distributed storage node and the target database. After normalization, these parameters can more accurately reflect the network status. For example, the bandwidth utilization coefficient can indicate the current network bandwidth usage, the delay attenuation coefficient reflects the degree of data transmission delay, and the packet loss penalty coefficient is used to measure the impact of packet loss on transmission.

[0055] Step 1343: Perform weighted calculation on the priority weight and the normalized network parameter to generate a comprehensive transmission priority score for each data slice, reorder the initial transmission sequence based on the score to generate an optimized transmission sequence, and adjust the transmission window size of each data slice in the transmission sequence according to the ratio of the current available bandwidth to the data slice size.

[0056] Optionally, the increased priority weight is weighted with the normalized network parameters. For example, a certain weight value is assigned to the priority weight, and corresponding weight values ​​are also assigned to the bandwidth utilization coefficient, delay attenuation coefficient, and packet loss penalty coefficient. Through weighted calculation, a comprehensive transmission priority score for each data slice is obtained. Based on this score, the initial transmission sequence is reordered, placing the data slices with higher priority at the front. At the same time, the transmission window size of each data slice is reasonably adjusted based on the ratio of the currently available bandwidth to the data slice size. For example, if a data slice is large and the currently available bandwidth is limited, its transmission window size is appropriately reduced to ensure that the network is not congested due to the transmission of one large slice.

[0057] Step 1344: Encapsulate the data fragments in the transmission sequence into data packets carrying fragment identifiers and check codes through the preset flow control strategy, start the parallel transmission channel, and monitor the transmission delay and packet loss rate changes of each data packet in real time during the transmission process.

[0058] Optionally, data in the transmission sequence is fragmented and encapsulated according to a preset flow control strategy. Each data packet carries a fragment identifier and a checksum, making it easy to identify and verify data during transmission. Parallel transmission channels are enabled to transmit multiple data packets simultaneously, improving transmission efficiency. During transmission, the system monitors the transmission delay and packet loss rate of each data packet in real time. For example, network monitoring tools can be used to obtain real-time information on the time required for each data packet to be transmitted from the distributed storage node to the target database, as well as any packet loss.

[0059] Step 1345: When the change in the packet loss rate of any data segment is monitored to exceed the preset tolerance threshold, the transmission window size of the data segment is reduced to a second preset level according to the flow control strategy, and the transmission window size of the adjacent data segment is simultaneously increased based on the current overall bandwidth occupancy to maintain the balance of bandwidth occupancy.

[0060] It can be understood that if the change in the packet loss rate of a certain data segment is monitored to exceed the preset tolerance threshold, for example, the preset tolerance threshold is 5%, and the change in the packet loss rate of the data segment reaches 8%. At this time, according to the flow control strategy, the transmission window size of the data segment will be reduced to the second preset level. At the same time, in order to maintain the balance of the overall bandwidth occupancy, the system will synchronously increase the transmission window size of the adjacent data segment according to the current overall bandwidth occupancy. For example, if the current overall bandwidth occupancy is 70%, when reducing the transmission window size of the data segment with an excessive packet loss rate, the transmission window size of its adjacent data segment will be increased accordingly, so that the overall bandwidth occupancy remains within a reasonable range, ensuring the stability and efficiency of data transmission.

[0061] Step 1346: After the target database returns a write confirmation signal carrying the shard identifier, the transmission log status of the corresponding shard identifier is updated; if the write confirmation signal of the target shard identifier is not received within the preset timeout period, the corresponding data shard is extracted from the distributed storage node based on the flow control strategy, repackaged, and inserted into the head of the transmission sequence for retransmission.

[0062] When the target database successfully receives a data shard and completes the write operation, it returns a write confirmation signal containing the shard identifier. Upon receiving this signal, the system immediately updates the transfer log status for the corresponding shard identifier, marking it as successfully transferred and written. For example, the transfer log records detailed information such as the transfer time and write time for the shard identifier.

[0063] However, if the system does not receive a write confirmation signal for the target shard identifier within a preset timeout period (for example, a 30-second timeout), if no confirmation signal is received after that time, it indicates that there may be a problem with the transmission of the data shard. In this case, based on the flow control strategy, the system extracts the corresponding data shard from the distributed storage node and repackages it to ensure data integrity and accuracy. The repackaged data shard is then inserted into the head of the transmission queue and retransmitted first to ensure that the data shard is successfully transmitted to the target database.

[0064] Step 135: During the transmission process, a target verification mechanism is used to perform a cyclic redundancy check on the data packet of each fragment. If the verification fails, the retransmission process of the corresponding fragment is executed until all fragments are completely written into the buffer queue of the target database.

[0065] During data transmission from distributed storage nodes to the target database, a target verification mechanism performs a cyclic redundancy check (CRC) on each shard's data packet to ensure data accuracy. CRC verifies data integrity by calculating a checksum on the data packet. For example, for each data packet, the system calculates a checksum value using a common algorithm and appends it to the packet. When the target database receives the packet, it recalculates the checksum value and compares it with the received checksum value.

[0066] If the comparison results indicate a verification failure, meaning the two checksum values ​​are unequal, this indicates a possible error in the data packet transmission. At this point, the system will retransmit the corresponding shard. As previously described, the data shard is re-extracted from the distributed storage node, repackaged, and transmitted again. This process is repeated until all shards pass verification and are fully written to the target database's buffer queue. This ensures that the data transmitted to the target database is accurate, providing a reliable foundation for subsequent data processing and use.

[0067] Step 140: Conflict detection and identification is performed based on the RPA conflict handling strategy configured in the target database. When a primary key conflict or a unique constraint conflict is identified, the target database is updated based on preset conflict merge indication information, and a data synchronization log is generated.

[0068] In the aforementioned database system, the target database has been configured with an RPA conflict resolution policy. When the target data set is transferred to the target database, the system uses this policy to detect and identify conflicts. For example, when inserting new product data, a primary key conflict may occur (i.e., the primary key of the newly inserted product matches the primary key of an existing product in the target database); or a unique constraint conflict may occur (i.e., if the unique identifier field for a product already has the same value in the database).

[0069] Once conflicts are identified, the system updates the target database according to pre-set conflict merge instructions. A data synchronization log is also generated to record conflicts throughout the data synchronization process. This log helps administrators understand issues that occurred during the data synchronization process and facilitates subsequent analysis and resolution.

[0070] In an alternative embodiment, step 140 includes: Step 141: configuring a conflict detection rule set in the target database, wherein the conflict detection rule set includes a primary key duplication detection rule, a foreign key constraint verification rule, and a unique index conflict rule.

[0071] In the target database of the aforementioned database system, the administrator pre-configures a set of conflict detection rules. These primary key duplication detection rules check whether the primary key of a newly inserted or updated data record duplicates an existing primary key in the database. For example, in a product information table, each product has a unique product ID as its primary key. This rule can detect whether a newly inserted product ID already exists.

[0072] Foreign key constraint validation rules are used to ensure the correctness of foreign key relationships. For example, the product ID in the order table is a foreign key that relates to the primary key of the product information table. This rule verifies whether the product ID referenced in the order table exists in the product information table to ensure data consistency.

[0073] Additionally, unique index conflict rules are used to detect duplicate values ​​in fields with unique indexes. For example, if a product's barcode field has a unique index, this rule can prevent the insertion of different product records with the same barcode. These rules provide comprehensive conflict detection for data in the target database.

[0074] Step 142: Create a data write transaction in the temporary storage area of ​​the target database, apply the conflict detection rule set in sequence to perform conflict scanning on the target data set, and generate a conflict data list.

[0075] When the target data set arrives at the target database, the system creates a data write transaction in the target database's temporary storage area. This transaction performs a series of operations on the target data set. The system then applies each rule in the conflict detection rule set sequentially, performing a comprehensive conflict scan on the target data set.

[0076] For example, the primary key duplication detection rule is first applied to check whether the primary key of each data record in the target data set is a duplicate of an existing primary key in the target database. Next, the foreign key constraint validation rule is applied to verify the correctness of foreign key relationships. Finally, the unique index conflict rule is applied to check for conflicts in fields with unique indexes. During the scanning process, all detected conflicting data is recorded and a conflict data list is generated. This list details the relevant information for each conflicting data, such as the table in which the conflict occurred, the fields involved, and the conflict type, providing a basis for subsequent conflict resolution.

[0077] Step 143: Match preset conflict merging indication information according to the conflict type in the conflict data list, the conflict merging indication information includes a timestamp overwriting strategy, a field priority strategy and a manual review strategy, and the conflict type includes a primary key conflict or a unique constraint conflict.

[0078] After generating a list of conflicting data, the system matches the pre-set conflict merge instructions based on the conflict types recorded in the list. For primary key conflicts, if a timestamp overwrite strategy is used, the source transaction timestamp of the conflicting data record is compared with the timestamp of the current record in the target database, and the record with the later timestamp is used for overwriting and updating. For example, if the source transaction timestamp of a newly inserted product record is newer than the timestamp of a product record with the same primary key in the target database, the new record overwrites the old one.

[0079] For unique constraint conflicts, if a field priority strategy is used, the preconfigured field priority mapping table will be combined to determine which of the conflicting fields has a higher weight. For example, if a conflict occurs in a product's unique identifier field, the field priority mapping table may indicate that the product name field has a higher weight. In this case, the data for the product name field will be retained, and adjustments will be made to other related fields.

[0080] If some complex or special conflict situations occur, a manual review strategy may be adopted to submit the conflicting data records to the administrator for manual judgment and processing to ensure that the conflict is resolved reasonably.

[0081] Step 144: The RPA robot executes the automated operation corresponding to the conflict merge indication information, updates the conflicting data records in the target database, and writes the update results into the data synchronization log.

[0082] As you can understand, after determining the conflict merge instructions, the RPA robot will perform the corresponding automated operations. For example, if the timestamp overwrite strategy is used, the RPA robot will follow the rules and overwrite the old record in the target database with the new record, completing the primary key conflict resolution. For the field priority strategy, the RPA robot will adjust and merge the data of the conflicting fields according to the field priority mapping table, and update the conflicting data records in the target database.

[0083] After completing these updates, the RPA robot writes detailed update results to the data synchronization log. The log records the update time, the tables and fields involved, the operations performed, and a comparison of the data before and after the update. Administrators can review the data synchronization log to clearly understand conflict resolution during the synchronization process, facilitating auditing and troubleshooting.

[0084] In an optional embodiment, step 144 includes: Step 1441: According to the conflict type in the conflict data list, extract the source transaction timestamp of the conflict data record and the current record timestamp of the target database, perform time unit standardization conversion based on the source transaction timestamp and the current record timestamp, and generate a time coverage priority parameter.

[0085] As you can understand, for each conflicting data record in the conflict data list, the system first determines the record to be processed based on the conflict type. If it is a primary key conflict or a timestamp-related unique constraint conflict, the system extracts the source transaction timestamp of the conflicting data record and the timestamp of the current record in the target database.

[0086] For example, for a product record with a primary key conflict, the source transaction timestamp is "2023-10-06 14:30:00," while the current record in the target database has a timestamp of "2023-10-05 11:45:00." To accurately compare these two timestamps, a time unit normalization conversion is required. These two timestamps are converted to a unified time representation, such as the number of seconds from a fixed start time. This normalization conversion generates a time overwrite priority parameter that clearly indicates which timestamp is "newer," providing a quantitative basis for subsequent timestamp overwrite strategies.

[0087] Step 1442: In combination with the preconfigured field priority mapping table, the normalized weight values ​​of the conflicting fields in the conflicting data records are extracted and field priority parameters are generated. The time coverage priority parameters and the field priority parameters are weighted and fused according to a preset ratio to generate a comprehensive priority sorting sequence for each conflicting data record.

[0088] When processing conflicting data records, if field priority policies are involved, the system will incorporate a preconfigured field priority mapping table. For example, for conflicting data records in a product information table, the conflicting fields might include product name, price, and inventory. The field priority mapping table determines the normalized weight for each conflicting field. For example, the normalized weight for product name is 0.6, the normalized weight for price is 0.3, and the normalized weight for inventory is 0.1.

[0089] These normalized weights are used as field priority parameters and weighted together with the previously generated time coverage priority parameters according to a preset ratio. For example, the preset time coverage priority parameter weight is 0.4, and the field priority parameter weight is 0.6. This weighted calculation generates a comprehensive priority ranking sequence for each conflicting data record. This sequence helps the system determine the processing priority of each conflicting data record, allowing for more reasonable conflict resolution.

[0090] Step 1443: Traverse the conflicting data records based on the comprehensive priority sorting sequence: if it is a primary key conflict and the comprehensive priority exceeds the preset priority, generate a target record replacement script and perform an overwrite update; if it is a unique constraint conflict and the priority does not reach the preset priority, generate a conflict field merge script based on the field priority comparison, and retain the weight field data.

[0091] After generating a comprehensive priority ranking sequence, the system will traverse the conflicting data records according to this sequence. For primary key conflicts, if the comprehensive priority of a conflicting data record exceeds the preset priority (for example, the preset priority is 0.5, and the comprehensive priority of the record is 0.6), this record requires priority processing and should be overwritten. The system will generate a target record replacement script, which is executed by the RPA robot to overwrite the old record in the target database with the new one.

[0092] For unique constraint conflicts, if the priority of a conflicting data record does not meet the preset priority, the system will generate a conflict field merging script based on field priority comparison. For example, if a conflict occurs in a product's unique identifier field, the system will use the field priority mapping table to determine the field with the higher weight. For example, if the product name field has a higher weight, the script will retain the data in the product name field and adjust and merge other related fields to resolve the unique constraint conflict.

[0093] Step 1444: After the script is executed, the RPA robot initiates a verification request containing a transaction identifier to the target database, and compares the actual data returned with the expected value of the script field by field. If there is a difference, the difference record is reinserted into the conflict data list and an alarm event is activated.

[0094] After the RPA robot executes the target record replacement script or the conflicting field merge script, it initiates a verification request containing a transaction identifier to the target database to ensure the accuracy of the update operation. The target database then returns the actual updated data.

[0095] The system compares the actual data returned against the script's expected values ​​field by field. For example, when updating product information, the actual and expected values ​​for each field, such as product name, price, and inventory, are compared. If a discrepancy is found, such as a discrepancy between the actual price and the expected price, the discrepancy record is reinserted into the conflicting data list for further analysis and resolution. Simultaneously, an alert is activated to notify administrators of the data inconsistency, requiring prompt attention and resolution.

[0096] Step 1445: Associate the execution result of the target record replacement script or the conflict field merging script, the field-by-field comparison result and the transaction identifier to generate a log entry with a version identifier; synchronously write the log entry into the data synchronization log partition of the distributed storage node and the audit table of the target database, and generate a reverse index relationship with the original data shard.

[0097] Optionally, the system associates the execution results of the target record replacement script or conflict field merging script, the field-by-field comparison results, and the transaction identifier. For example, the system can combine the updated product information record after script execution, the differences discovered during the comparison, and the corresponding transaction identifier to generate a log entry with a version identifier. This version identifier can be used to distinguish different versions of the operation record, facilitating traceability and management.

[0098] The log entry is then synchronously written to the data synchronization log partition of the distributed storage node and the audit table of the target database. The data synchronization log partition records detailed information about the entire data synchronization process, facilitating administrators' queries and analysis. The audit table of the target database can be used for auditing and compliance checks. Furthermore, to facilitate tracing back from log records to the original data shards, a reverse index relationship with the original data shards is generated. For example, by recording the identifier of the original data shard in the log entry, a link is established between the log and the original data, allowing administrators to quickly locate and view relevant original data when needed.

[0099] In another embodiment, the method further includes step 210 , step 220 , step 230 or step 240 .

[0100] Step 210: During the data synchronization process, the running status of the distributed computing nodes is monitored in real time. When a node failure or task timeout is detected, the unfinished tasks are automatically reallocated to healthy nodes.

[0101] During the data synchronization process of the aforementioned database system, the system monitors the operating status of the distributed computing nodes in real time. It determines whether the nodes are operating normally by monitoring indicators such as CPU utilization, memory utilization, and network connection status. For example, if a computing node's CPU utilization consistently exceeds 90%, its memory utilization approaches 100%, and its network connection is unstable, this may indicate that the node is about to fail.

[0102] Alternatively, if a data sharding task fails to complete within the preset timeframe, it is considered a task timeout. For example, if the preset completion time is 60 minutes and a task remains incomplete after 90 minutes, the system will automatically reassign the unfinished task from the problematic node to a healthy node. For example, an unfinished data sharding task can be transferred from a failed node to a healthy node with lower CPU and memory usage and a stable network connection. This ensures that the data synchronization task can be completed smoothly without being interrupted by a single node problem.

[0103] Step 220: If a format parsing error occurs during the data conversion process, the RPA robot calls the backup conversion rule set to retry, and extracts the erroneous data segment after the retry fails and sends it to the repair queue.

[0104] During the format conversion process of the initial shards by the distributed computing nodes, format parsing errors may occur. For example, when converting unstructured data to the target structured data block, the pre-trained NLP model may not correctly parse certain text content, causing the format conversion to fail.

[0105] When this happens, the RPA robot will retry by using an alternative conversion rule set. This alternative conversion rule set might use a different algorithm or model to convert the data format, hoping to successfully process the data. For example, while the original conversion rule set uses a machine learning-based approach, the alternative conversion rule set might use a rule engine-based approach.

[0106] If the retry still fails, the system extracts the error data segment. For example, if the product description text cannot be converted correctly, this segment will be extracted as the error data segment. This error data segment is then sent to the repair queue. The repair queue is used to process error data. Administrators or subsequent processing processes can retrieve the error data from this queue for further analysis and repair.

[0107] Step 230: When a connection interruption occurs during the writing process of the target database, the breakpoint position is automatically recorded and the writing operation is continued from the breakpoint position after the connection is restored.

[0108] During the process of writing the target data set to the target database, the connection may be interrupted due to network failure, database server failure, etc. For example, when writing product information data, the network may be suddenly interrupted, resulting in some data not being successfully written.

[0109] When this happens, the system automatically records the breakpoint location. For example, it records the location of the last data that was successfully written and the related information of the data that should be written next. The breakpoint location can be represented by data offset, record number, etc.

[0110] When the connection is restored, the system will resume writing from the breakpoint. It will find the unwritten data based on the previously recorded breakpoint information and continue writing it to the target database. For example, if 100 product information records have been successfully written and the breakpoint is recorded as the 100th record, after the connection is restored, the system will continue writing from the 101st record to ensure data integrity and continuity.

[0111] Step 240: Periodically analyze the error patterns in the data synchronization log, and adjust the data sharding task allocation strategy and conflict detection rule set to optimize synchronization efficiency.

[0112] The system periodically analyzes data synchronization logs, for example, once every morning. The data synchronization logs contain various error messages, such as node failures, format parsing errors, and conflict resolution failures. By analyzing these error messages, we can extract error patterns.

[0113] For example, you might discover frequent format parsing errors due to formatting issues with certain fields within a certain time period, or that a data sharding task frequently times out on certain nodes. Based on these error patterns, adjust the data sharding task allocation strategy. If a data sharding task is prone to issues on certain nodes, adjust the allocation strategy to avoid assigning the task to those nodes.

[0114] At the same time, the conflict detection rule set is adjusted based on error patterns. For example, if a certain type of conflict is found to occur frequently and be handled inefficiently, the rules in the conflict detection rule set that target that type of conflict are optimized. For example, more detailed pre-check conditions can be added, or parameters in the conflict merge indication information can be adjusted. This allows conflicts to be detected and handled more quickly and accurately, thereby optimizing the efficiency of the entire data synchronization process, reducing errors, and improving the stability and reliability of data synchronization.

[0115] As a technical solution that can be implemented independently, after the target database is updated based on the preset conflict merge indication information and the data synchronization log is generated, it also includes: periodically collecting the conflict type distribution frequency and task execution time interval in the data synchronization log, converting the conflict type distribution frequency into a conflict weight vector according to a preset normalization rule, and performing a difference calculation between the task execution time interval and the preset reference time parameter to generate a time deviation coefficient; inputting the conflict weight vector and the time deviation coefficient into a pre-trained synchronization strategy optimization model, and outputting a sharding rule adjustment coefficient and a conflict detection sensitivity parameter; updating the sharding size threshold of the data sharding task based on the sharding rule adjustment coefficient, and adjusting the priority weight of the timestamp overlay strategy in the RPA conflict handling strategy according to the conflict detection sensitivity parameter; generating the data sharding task allocation instruction and conflict merging rule sequence for the next cycle based on the updated sharding size threshold and priority weight, and injecting them into the RPA robot configuration file of the distributed computing node and the target database.

[0116] During the data synchronization process within the aforementioned database system, the system periodically analyzes the data synchronization logs to continuously optimize the synchronization strategy. For example, data collection is performed weekly. From the data synchronization logs, two key pieces of information are carefully collected: the frequency distribution of conflict types and the interval between task executions.

[0117] For conflict type distribution frequencies, record the number of times different conflict types (such as primary key conflicts and unique constraint conflicts) occur over a period of time. For example, suppose that in the past week, primary key conflicts occurred 20 times, unique constraint conflicts occurred 15 times, and so on. These conflict type distribution frequencies are then converted into a conflict weight vector using a pre-set normalization rule. The normalization rule might be to divide the frequency of each conflict type by the total number of conflicts to determine the relative weight of each conflict type. For example, if the total number of conflicts is 35, the weight of the primary key conflict is 20 / 35, the weight of the unique constraint conflict is 15 / 35, and so on. This results in a vector that clearly reflects the relative importance of various conflict types in the data synchronization process.

[0118] At the same time, task execution time interval information is collected. The time taken for each data sharding task from start to completion is calculated and compared with a preset reference time parameter. For example, if the preset reference time parameter is 30 minutes and the actual execution time of a data sharding task is 40 minutes, a time deviation coefficient is calculated by difference. An exemplary time deviation coefficient is 40-30 = 10 minutes (those skilled in the art may consider various additional factors based on actual needs).

[0119] The generated conflict weight vector and time deviation coefficient are fed into a pre-trained synchronization strategy optimization model. This model, trained on a large amount of data, is capable of intelligently analyzing and processing the input information. The model outputs the sharding rule adjustment coefficient and the conflict detection sensitivity parameter.

[0120] The shard size threshold for data sharding tasks is updated based on the sharding rule adjustment coefficient. For example, if the sharding rule adjustment coefficient indicates that the current shard size may lead to low data processing efficiency and requires an appropriate increase in the shard size, the shard size threshold is increased accordingly. For example, if the original shard size threshold is based on 1000 records per shard, the adjustment coefficient may adjust the threshold to 1500 records per shard.

[0121] Adjust the priority weight of the timestamp overwrite strategy within the RPA conflict handling strategy based on the conflict detection sensitivity parameter. If the conflict detection sensitivity parameter indicates that timestamp accuracy should be prioritized, increase the priority weight of the timestamp overwrite strategy within the overall conflict handling strategy. For example, if the original timestamp overwrite strategy priority weight is 0.4, adjust it to 0.6 based on the parameter.

[0122] Finally, based on the updated shard size threshold and priority weights, the next cycle's data sharding task allocation instructions and conflict merging rule sequence are generated. These instructions and rule sequences are injected into the RPA robot configuration files of the distributed computing nodes and the target database. During the next cycle of data synchronization, the distributed computing nodes and RPA robots will operate according to the new policies and rules, continuously optimizing data synchronization efficiency and accuracy.

[0123] As a technical solution that can be implemented independently, after the target database is updated based on the preset conflict merge indication information and a data synchronization log is generated, it also includes: extracting data blocks from the target data set written in the target database according to the shard identifier, calculating the hash summary value of each data block and generating a shard hash set; obtaining the corresponding initial data set from the distributed storage node, generating a source shard hash set according to the same shard identifier and hash algorithm; comparing the shard hash set with the source shard hash set item by item, and if there are shard identifiers with inconsistent hash summary values, generating a difference shard identification list and performing a data repair task; extracting the original data shards from the distributed storage node according to the difference shard identification list, re-executing the format conversion operation to generate repair data shards, and writing the repair data shards into the independent repair storage area of ​​the target database through the data synchronization channel based on the RPA conflict handling strategy; after the repair data shards are written, updating the repair status mark in the data synchronization log, and writing the mapping relationship between the difference shard identification list and the physical address of the repair storage area into the reverse index table.

[0124] After data synchronization is complete, to ensure the accuracy and consistency of the data in the target database, an integrity check is performed on the target data set already written to the target database. First, data blocks are extracted from the target database according to the shard identifier. For example, for product information data, data from different shards is extracted based on the previously assigned shard identifiers.

[0125] Then, for each extracted data block, a hash digest value is calculated. The hash algorithm can be a widely used algorithm, such as SHA-256. By hashing the data block, a unique hash digest value is generated, which can represent the content characteristics of the data block. The hash digest values ​​of all data blocks are collected to form a shard hash set.

[0126] At the same time, the corresponding initial data set is obtained from the distributed storage node. Using the same shard identifier and hash algorithm, each shard data block in the initial data set is hashed to generate the source shard hash set.

[0127] Next, the shard hash set is compared item by item with the source shard hash set. The hash digest values ​​of the data blocks corresponding to the same shard identifier in the two sets are compared one by one. If a shard identifier with an inconsistent hash digest value is found, it indicates that there may be a problem with the data in that shard during the synchronization process. For example, if the hash digest value of the data block in the target database corresponding to a shard identifier differs from the hash digest value of the original data block in the distributed storage node, the shard identifier is recorded and a list of different shard identifiers is generated.

[0128] Once the differential shard identifier list is generated, the data repair task is executed. Based on this list, the original data shards are retrieved from the distributed storage nodes. For example, based on a shard identifier in the differential shard identifier list, the corresponding original data shard is accurately found from the distributed storage node.

[0129] Then, the format conversion operation is re-performed on the extracted original data shards. According to the previous data conversion process, the shard data is aligned with the structured data schema and the unstructured data semantics is normalized to generate the repaired data shards.

[0130] Based on the RPA conflict resolution strategy, the repaired data shards are written to the independent repair storage area of ​​the target database through the data synchronization channel. During the writing process, the data synchronization process and rules are followed to ensure that the data is written accurately.

[0131] When the repaired data shard is written, the repair status flag in the data synchronization log is updated. This log clearly records that the shard data has been repaired and successfully written, facilitating subsequent queries and audits. Simultaneously, the mapping between the difference shard identifier list and the physical address of the repair storage area is written to a reverse index table. This reverse index table helps administrators quickly locate and find the location of the problematic data shard in the repair storage area, allowing for further inspection and processing when necessary. This series of operations ensures that the data in the target database remains accurate and complete, improving data reliability and availability.

[0132] In actual application, technical details can be supplemented and processes can be optimized based on existing database log parsing technology, distributed computing framework and NLP model application experience.

[0133] For example, to address the insufficient disclosure of data extraction scripts generated by RPA robots, existing dynamic SQL generation technology can be used to automatically match operation parameters with script templates. This, combined with an operation type mapping rule library (such as MyBatis's dynamic SQL mechanism), allows conditional statement assembly, ensuring the feasibility of script generation. For NLP semantic parsing of unstructured data, the entity recognition interface of existing open source pre-trained models (such as BERT) can be introduced. By configuring entity extraction rules and relationship graphs to build templates, key entities and contextual relationships in the text can be converted into structured fields.

[0134] To quantify sharding rules, we can leverage existing distributed database sharding strategies (such as Cassandra's consistent hashing algorithm) and use table association graph analysis tools to calculate the frequency of inter-table connections. This, combined with field dependency detection algorithms (such as functional dependency inference), can dynamically generate sharding weight coefficients to quantitatively assess data distribution characteristics. For parameter calculation in flow control strategies, we can draw on the sliding window mechanism of the TCP congestion control algorithm. Using existing network quality probe tools, we can obtain parameters such as bandwidth, latency, and packet loss rate in real time. These parameters can then be normalized using an exponentially weighted moving average algorithm to establish a dynamic priority adjustment model.

[0135] During timestamp standardization, ISO8601 time format conversion technology can be used, combined with a time zone offset compensation algorithm, to achieve unified time bases across systems. For field priority mapping in conflict detection, a field impact assessment model can be constructed using existing data lineage analysis tools. This, combined with the business rule engine to configure field weight coefficients, ensures the configurability of conflict merging strategies.

[0136] This design can fully implement the entire process from data change monitoring, distributed processing to conflict resolution.

[0137] The embodiment of the present application monitors data changes through the RPA robot deployed on the source database server, which can capture changes in structured and unstructured data in real time, generate the initial data set in a timely manner, and ensure the timeliness and integrity of the data. The initial data set is fragmented and distributed to the distributed computing nodes to perform parallel format conversion, which greatly improves the data processing efficiency, fully utilizes the advantages of distributed computing, and quickly generates the target data set. By detecting the execution status of the distributed computing node tasks in real time, the data synchronization channel is activated once the target data set is generated, and the data can be accurately and efficiently transmitted to the target database to ensure the timeliness of data flow. Conflict detection and identification are performed in the target database based on the configured RPA conflict handling strategy, and the database is updated and synchronization logs are generated according to the preset information. This can effectively solve the conflict problem in data synchronization, ensure the consistency and accuracy of the data, improve the reliability and stability of the entire data synchronization process, and provide solid data support for the stable operation of the RPA business system.

[0138] In summary, the embodiments of the present application effectively improve the timeliness, efficiency and consistency of data processing through the real-time monitoring of data changes by RPA robots, distributed parallel processing and precise conflict detection and processing mechanisms.

[0139] It is important to note that with respect to the data acquisition and processing processes involved in the embodiments of this application, all data operations are performed with the user's full knowledge and explicit authorization. During user registration and service usage, the system informs the user of the data collection scope, usage purpose, and sharing scenarios through an explicit privacy agreement. The system then obtains the user's specific authorization for the collection, processing, and synchronization of relevant data, such as product information and order records, by selecting a checkbox. For operations involving sensitive personal data (such as modifying order addresses), an enhanced authorization mechanism is implemented, confirming the user's intent through secondary verification. During data synchronization, RPA robots only process desensitized, non-identity-related data. Key fields are protected end-to-end using the AES-256 encryption algorithm, and all data change event monitoring is restricted to the security boundaries of the enterprise's internal authorization server. The system regularly conducts data security impact assessments, controls data access levels based on the principle of minimal privilege, and maintains a comprehensive audit log of data operations for compliance review. Users can view authorization scope, revoke data sharing permissions, or initiate data deletion requests in real time through the enterprise service portal. The system will complete authorization status synchronization and data cleanup within a few business days, effectively safeguarding the user's right to know and data autonomy.

[0140] See also Figure 2 As shown, this figure is a schematic diagram of the basic structure of a database multi-source heterogeneous data synchronization device 200 provided in an embodiment of the present application. The database multi-source heterogeneous data synchronization device 200 includes: Processor 201; a storage device 202 having a computer program 2020 stored thereon; When the computer program 2020 is executed by the processor 201, the processor 201 implements any of the database multi-source heterogeneous data synchronization methods.

[0141] Based on the above, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the above method are implemented.

[0142] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems or devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

Claims

1. A method for synchronizing multi-source heterogeneous data in a database, characterized in that: include: An RPA robot deployed on the server where the source database resides monitors data change events and generates an initial data set when a change operation on structured or unstructured data is detected. Slicing the initial data set to generate a plurality of data slicing tasks, and assigning the data slicing tasks to distributed computing nodes to perform parallel format conversion operations to generate a target data set; detecting the task execution status of the distributed computing node, and when detecting a generation completion event of the target data set, activating a data synchronization channel and transmitting the target data set to a target database; Conflict detection and identification are performed based on the RPA conflict handling strategy configured in the target database. When a primary key conflict or a unique constraint conflict is identified, the target database is updated based on preset conflict merge indication information, and a data synchronization log is generated.

2. The database multi-source heterogeneous data synchronization method according to claim 1, characterized in that: The RPA robot deployed on the server where the source database is located monitors data change events. When a change operation on structured or unstructured data is detected, an initial data set is generated, including: The RPA robot parses the operation log of the source database in real time to extract a set of event records containing data insertion, update, or deletion operations; Performing a comparison analysis based on the transaction timestamps in the event record set and the previous synchronization time of the target database, filtering out unsynchronized transaction operation records from the event record set based on the comparison analysis results, and generating a target transaction operation set; Simulating a database query operation by the RPA robot, generating a data extraction script according to the operation type and operation parameters in the target transaction operation set, and executing the data extraction script to pull the initial data set from the source database; Add transaction identifier and timestamp metadata to the initial data set, and cache the initial data set to a distributed storage node.

3. The database multi-source heterogeneous data synchronization method according to claim 2, characterized in that: The sharding process is performed on the initial data set to generate a plurality of data sharding tasks, and the data sharding tasks are distributed to distributed computing nodes to perform parallel format conversion operations to generate a target data set, including: Based on the data distribution characteristics of the initial data set, the initial data set is divided into a plurality of initial shards using preset sharding rules, wherein the sharding rules include data table association, data volume balance, and field dependency; Creating an independent data conversion task for each initial shard, and assigning the data conversion task to an idle computing node based on the load status of the distributed computing node; In each distributed computing node, a pre-configured format conversion rule set is called to perform structured data schema alignment and unstructured data semantic standardization on the allocated initial shards to generate intermediate standardized shards. Aggregate all intermediate standardized shards to obtain shard aggregation results, perform data integrity verification and deduplication processing on the shard aggregation results, and generate the target data set.

4. The database multi-source heterogeneous data synchronization method according to claim 3, characterized in that: In each distributed computing node, a preconfigured format conversion rule set is called to perform structured data schema alignment and unstructured data semantic standardization on the allocated initial shards to generate intermediate standardized shards, including: Perform field mapping matching on the structured data tables in the initial shards. Use RPA scripts to automatically compare source field names with the field definitions in the target database and generate a field mapping relationship table. Performing automatic type conversion including string encoding conversion, date format unification, and numerical precision adjustment on fields with inconsistent data types according to the field mapping relationship table to obtain a converted structured data table; Perform semantic parsing on the unstructured data blocks in the initial shards, extract key entities and contextual relationships through a pre-trained NLP model, and convert the unstructured data into target structured data blocks containing entity labels and relationship graphs; The converted structured data table and the target structured data block are reorganized according to the storage rules of the target database and a shard identifier and a check code are added to generate an intermediate standardized shard.

5. The database multi-source heterogeneous data synchronization method according to claim 1, characterized in that: The detecting the task execution status of the distributed computing node, and when detecting the target data set generation completion event, activating the data synchronization channel and transmitting the target data set to the target database, includes: The RPA task status collector obtains the execution status parameters of each data shard task in the distributed computing node in real time. The execution status parameters include at least the task progress percentage and the exception error code. When it is identified that the progress percentages of all data sharding tasks have reached the preset percentage and the exception error code is empty, a target data set generation completion event is generated and the data synchronization channel run instruction is activated; According to the data synchronization channel running instruction, a pre-configured channel connector is called to establish a bidirectional encrypted channel with the target database, and the channel connector loads a corresponding communication protocol driver based on the type of the target database; Arrange the target data set in the distributed storage nodes in the order of shard identifiers for transmission and inject a flow control strategy to optimize network congestion; During the transmission process, the target verification mechanism is used to perform cyclic redundancy check on the data packet of each fragment. If the verification fails, the retransmission process of the corresponding fragment is executed until all fragments are completely written into the buffer queue of the target database.

6. The database multi-source heterogeneous data synchronization method according to claim 5, characterized in that: The target data set in the distributed storage node is arranged in a transmission sequence according to the order of the shard identifiers, and a flow control strategy for optimizing network congestion is injected, including: Parse the shard identifier of each data shard in the target data set, extract the global sequence number field and the shard generation timestamp, generate the initial transmission sequence according to the ascending value of the global sequence number field, and calculate the waiting transmission time of each data shard based on the difference between the shard generation timestamp and the current system timestamp; When the waiting transmission time exceeds the preset transmission time threshold, the priority weight of the corresponding data shard is increased to the first preset level, and a real-time network parameter set between the distributed storage node and the target database is synchronously collected. The network parameter set includes a normalized bandwidth utilization coefficient, a delay attenuation coefficient, and a packet loss penalty coefficient; Performing a weighted calculation on the priority weights and the normalized network parameters to generate a comprehensive transmission priority score for each data slice, reordering the initial transmission sequence based on the score to generate an optimized transmission sequence, and adjusting the transmission window size of each data slice in the transmission sequence based on the ratio of the currently available bandwidth to the data slice size; The data fragments in the transmission sequence are encapsulated into data packets carrying fragment identifiers and checksums through a preset flow control strategy. Parallel transmission channels are started, and the transmission delay and packet loss rate changes of each data packet are monitored in real time during the transmission process. When the packet loss rate change of any data segment is detected to exceed a preset tolerance threshold, the transmission window size of the data segment is reduced to a second preset level according to the flow control strategy, and the transmission window size of the adjacent data segment is simultaneously increased based on the current overall bandwidth occupancy to maintain a balance in bandwidth occupancy; After the target database returns a write confirmation signal carrying the shard identifier, the transmission log status of the corresponding shard identifier is updated; if the write confirmation signal of the target shard identifier is not received within the preset timeout period, the corresponding data shard is extracted from the distributed storage node based on the flow control strategy, repackaged, and inserted into the head of the transmission sequence for retransmission.

7. The database multi-source heterogeneous data synchronization method according to claim 1, characterized in that: The conflict detection and identification is performed based on the RPA conflict handling strategy configured in the target database. When a primary key conflict or a unique constraint conflict is identified, the target database is updated based on preset conflict merge indication information, and a data synchronization log is generated, including: Configuring a conflict detection rule set in the target database, wherein the conflict detection rule set includes a primary key duplication detection rule, a foreign key constraint verification rule, and a unique index conflict rule; Creating a data write transaction in a temporary storage area of ​​a target database, applying the conflict detection rule set in sequence to perform conflict scanning on the target data set, and generating a conflict data list; Matching preset conflict merging indication information according to the conflict type in the conflict data list, the conflict merging indication information including a timestamp overwriting strategy, a field priority strategy, and a manual review strategy, the conflict type including a primary key conflict or a unique constraint conflict; The RPA robot executes the automated operation corresponding to the conflict merge indication information, updates the conflicting data records in the target database, and writes the update results into the data synchronization log.

8. The database multi-source heterogeneous data synchronization method according to claim 7, characterized in that: The step of executing the automated operation corresponding to the conflict merge indication information by the RPA robot, updating the conflicting data record in the target database, and writing the update result into the data synchronization log includes: Extracting the source transaction timestamp of the conflicting data record and the current record timestamp of the target database according to the conflict type in the conflict data list, performing time unit standardization conversion based on the source transaction timestamp and the current record timestamp, and generating a time coverage priority parameter; Combined with the pre-configured field priority mapping table, the normalized weight values ​​of the conflicting fields in the conflicting data records are extracted and the field priority parameters are generated. The time coverage priority parameter and the field priority parameter are weighted and fused according to a preset ratio to generate a comprehensive priority ranking sequence for each conflicting data record; Traverse the conflicting data records based on the comprehensive priority sorting sequence: if it is a primary key conflict and the comprehensive priority exceeds the preset priority, generate a target record replacement script and perform an overwrite update; if it is a unique constraint conflict and the priority does not reach the preset priority, generate a conflict field merging script based on field priority comparison, and retain the weight field data; After the script is executed, the RPA robot initiates a verification request containing a transaction identifier to the target database, compares the actual data returned with the expected value of the script field by field, and if there is a discrepancy, reinserts the discrepancy record into the conflict data list and activates an alarm event. The execution result of the target record replacement script or the conflict field merging script, the field-by-field comparison result and the transaction identifier are associated to generate a log entry with a version identifier; the log entry is synchronously written into the data synchronization log partition of the distributed storage node and the audit table of the target database, and a reverse index relationship with the original data shard is generated.

9. The database multi-source heterogeneous data synchronization method according to claim 1, characterized in that: The method further comprises: Monitor the running status of distributed computing nodes in real time during data synchronization. When a node failure or task timeout is detected, unfinished tasks are automatically reallocated to healthy nodes. Alternatively, if a format parsing error occurs during data conversion, the RPA robot calls the backup conversion rule set to retry, and extracts the erroneous data fragment after the retry fails and sends it to the repair queue; Alternatively, if a connection is interrupted during writing to the target database, the breakpoint location is automatically recorded and the writing operation is continued from the breakpoint location after the connection is restored; Alternatively, periodically analyze the error patterns in the data synchronization log and adjust the data sharding task allocation strategy and conflict detection rule set to optimize synchronization efficiency.

10. A database multi-source heterogeneous data synchronization device, characterized in that: include: processor; A storage device stores a computer program thereon, and when the computer program is executed by the processor, the processor implements the database multi-source heterogeneous data synchronization method as described in any one of claims 1 to 9.

Images