Database multi-source heterogeneous data synchronization method and device

Abstract

This application discloses a method and apparatus for synchronizing heterogeneous data from multiple sources in a database. The method includes: monitoring data change events by deploying an RPA robot on the server where the source database is located; generating an initial data set when a change operation on structured or unstructured data is detected; performing sharding processing on the initial data set to generate multiple data sharding tasks, and allocating the data sharding tasks to distributed computing nodes to perform parallel format conversion operations to generate a target data set; detecting the task execution status of the distributed computing nodes; activating a data synchronization channel and transmitting the target data set to the target database when a completion event of the target data set generation is detected; and performing conflict detection and identification based on the RPA conflict handling strategy configured in the target database. When a primary key conflict or uniqueness constraint conflict is identified, updating the target database based on preset conflict merging indication information and generating a data synchronization log.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a method and apparatus for synchronizing multi-source heterogeneous data in a database. Background Technology

[0002] With the rapid development of information technology, RPA database technology is playing an increasingly important role in the field of data processing and management. RPA can simulate human operations, realize automated processes, and improve work efficiency and accuracy. In terms of database management, it provides new solutions for tasks such as data migration and synchronization.

[0003] Under the current RPA database technology context, existing technologies still have the following shortcomings: the monitoring of data changes in the source database is not timely or comprehensive enough, and it cannot simultaneously take into account changes in structured and unstructured data, resulting in some data not being updated in a timely manner and affecting data integrity; during data processing, the data format conversion efficiency is low, and there is a lack of parallel processing capabilities, which cannot meet the needs of rapid processing of large-scale data; moreover, when synchronizing data to the target database, there is a lack of effective detection and handling mechanisms for potential conflicts, which can easily lead to data inconsistency and affect the normal operation of business systems. Summary of the Invention

[0004] This application provides a method and apparatus for synchronizing heterogeneous data from multiple database sources. It is used to effectively improve the timeliness, efficiency, and consistency of data processing by using an RPA robot to monitor data changes in real time, employing distributed parallel processing, and implementing a precise conflict detection and handling mechanism.

[0005] In a first aspect, embodiments of this application provide a method for synchronizing heterogeneous data from multiple database sources, applied to a device for synchronizing heterogeneous data from multiple database sources. The method includes: monitoring data change events by deploying an RPA robot on a server where the source database is located; generating an initial data set when a change operation on structured or unstructured data is detected; performing sharding processing on the initial data set to generate multiple data sharding tasks, and allocating the data sharding tasks to distributed computing nodes to perform parallel format conversion operations to generate a target data set; detecting the task execution status of the distributed computing nodes; activating a data synchronization channel and transmitting the target data set to the target database when a completion event of the generation of the target data set is detected; and performing conflict detection and identification based on the RPA conflict handling strategy configured in the target database. When a primary key conflict or uniqueness constraint conflict is identified, updating the target database based on preset conflict merging indication information and generating a data synchronization log.

[0006] Secondly, embodiments of this application provide a database multi-source heterogeneous data synchronization device, comprising:

[0007] processor;

[0008] Storage device, on which computer programs are stored,

[0009] When the computer program is executed by the processor, the processor implements any of the aforementioned database multi-source heterogeneous data synchronization methods.

[0010] This application provides a readable storage medium storing a program or instructions, which, when executed by a processor, implement the steps of the database multi-source heterogeneous data synchronization method.

[0011] Therefore, the embodiments of this application have the following beneficial effects: By deploying an RPA robot on the source database server to monitor data changes, changes in structured and unstructured data can be captured in real time, and an initial data set can be generated in a timely manner, ensuring the timeliness and integrity of the data. The initial data set is sharded and distributed to distributed computing nodes for parallel format conversion, greatly improving data processing efficiency and fully utilizing the advantages of distributed computing to quickly generate the target data set. By monitoring the task execution status of the distributed computing nodes in real time, the data synchronization channel is activated once the target data set is generated, enabling accurate and efficient transmission of data to the target database, ensuring the timeliness of data flow. Conflict detection and identification are performed in the target database based on the configured RPA conflict handling strategy, and the database is updated and synchronization logs are generated according to preset information. This effectively solves the conflict problem in data synchronization, ensures data consistency and accuracy, improves the reliability and stability of the entire data synchronization process, and provides solid data support for the stable operation of the RPA business system. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating a method for synchronizing heterogeneous data from multiple database sources, as provided in an embodiment of this application.

[0013] Figure 2 This is a schematic diagram of the basic structure of a database multi-source heterogeneous data synchronization device provided in an embodiment of this application. Detailed Implementation

[0014] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the embodiments of this application will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0015] See Figure 1 As shown, this figure is a flowchart of a database multi-source heterogeneous data synchronization method provided in an embodiment of this application. This method can be applied to a database multi-source heterogeneous data synchronization device. Figure 1As shown, the method may include steps 110-140.

[0016] Step 110: Listen for data change events by deploying an RPA robot on the server where the source database is located. When a change operation on structured or unstructured data is detected, an initial data set is generated.

[0017] In this embodiment, a database system of a large e-commerce company is used as an example. The source database of this system stores various types of product information, order information, and other data. An RPA robot deployed on the server hosting the source database is constantly monitoring. For example, when a new product is entered into the system (a structured data insertion operation), or the price of a product is modified (a structured data update operation), or a deleted order record is removed (an unstructured data deletion operation), the RPA robot will come into play and generate an initial data set. This initial data set contains the specific data content that has changed in the database, providing raw materials for subsequent data synchronization.

[0018] As an optional embodiment, step 110 includes:

[0019] Step 111: The RPA robot performs real-time parsing of the operation logs of the source database to extract a set of event records containing data insertion, update, or deletion operations.

[0020] In the above application scenario, the source database records logs for every operation. The RPA robot reads these operation logs in real time and parses them in an orderly and comprehensive manner. For example, the operation logs record information such as time, operator, operation type, and the data tables involved. The RPA robot carefully filters out records where the operation type is data insertion, update, or deletion. For instance, if the operation log shows that user A inserted a new product record into the product information table at a certain moment, the RPA robot will extract this record and add it to the event log set. In this way, all relevant operation records are collected to form a complete event log set, providing an accurate data source for subsequent processing.

[0021] Step 112: Compare and analyze the transaction timestamps in the event record set with the preceding synchronization time of the target database. Based on the comparison and analysis results, filter out the unsynchronized transaction operation records from the event record set to generate the target transaction operation set.

[0022] In the aforementioned database system, the target database has already undergone one synchronization operation, and the completion time of this operation is recorded as the previous synchronization time. The RPA robot extracts the transaction timestamp of each record in the event record set and compares it with the previous synchronization time. For example, if there is a product update record in the event record set whose transaction timestamp shows the time after the most recent synchronization, then this record belongs to the unsynchronized transaction operation record. By comparing them one by one, all such unsynchronized records are filtered out to form the target transaction operation set. The records in this set are all those that need to be synchronized subsequently, ensuring that no new change operations are missed.

[0023] Step 113: Simulate database query operations using the RPA robot, generate a data extraction script based on the operation type and operation parameters in the target transaction operation set, and execute the data extraction script to pull the initial data set from the source database.

[0024] For the aforementioned database system, the target transaction operation set contains various types of operation records. The RPA robot generates corresponding data extraction scripts based on the operation type and parameters of these records. For example, if it's a product information insertion operation, the operation parameters might include product ID, name, price, etc. The RPA robot generates a data extraction script based on this information to extract complete product information from the source database. Then, the RPA robot executes this script (which can be understood as manually executing query statements in the database), accurately pulling the initial data set related to these operations from the source database. This initial data set contains detailed product information, such as product description and inventory quantity, providing a comprehensive data foundation for subsequent data processing.

[0025] Step 114: Add transaction identifiers and timestamp metadata to the initial data set, and cache the initial data set to a distributed storage node.

[0026] In the aforementioned application scenario, to better track and manage the data synchronization process, the RPA robot adds a transaction identifier and timestamp metadata to each data record in the initial dataset. The transaction identifier uniquely identifies a data change operation, while the timestamp metadata records the specific time the operation occurred. For example, for a product update record, a transaction identifier "TX001" and a timestamp "2023-10-05 10:30:00" are added. Afterward, the initial dataset with this metadata is cached on distributed storage nodes. Distributed storage nodes can be considered a data warehouse; multiple storage nodes can jointly store and manage the data, ensuring data security and accessibility, and facilitating subsequent data sharding and other operations.

[0027] Step 120: The initial data set is sharded to generate multiple data sharding tasks, and the data sharding tasks are assigned to distributed computing nodes to perform parallel format conversion operations to generate the target data set.

[0028] Given the massive data volume of the aforementioned database system, sharding the initial data set is essential. In this embodiment, the initial data set is divided into multiple parts according to certain rules, forming multiple data sharding tasks. For example, product information data is sharded according to different product categories; data on electronic products is divided into one shard, and data on clothing is divided into another. These data sharding tasks are then distributed across distributed computing nodes, with each node simultaneously performing format conversion operations to improve processing efficiency. Finally, a target data set is generated, which, after format conversion, conforms to the storage requirements of the target database.

[0029] As another optional embodiment, step 120 includes:

[0030] Step 121: Based on the data distribution characteristics of the initial data set, the initial data set is divided into multiple initial shards using preset sharding rules. The sharding rules include data table correlation, data volume balance, and field dependency.

[0031] In the aforementioned database system, various relationships exist between different data tables, with varying data volumes and dependencies between fields. For example, there is a relationship between the product information table and the order table; one product may correspond to multiple orders. The preset sharding rules comprehensively consider these factors. Regarding table relationships, closely related data will be grouped into the same initial shard, such as grouping all product information and related order information for a single brand together. In terms of data volume balance, efforts will be made to ensure that the data volume in each initial shard is roughly equal to avoid excessively large data volumes in any one shard leading to excessively long processing times. Regarding field dependencies, such as the price field of a product depending on the product category field, data in related fields will be grouped into the same shard. Through these rules, the initial dataset is rationally divided into multiple initial shards.

[0032] Step 122: Create an independent data transformation task for each initial shard, and allocate the data transformation task to idle computing nodes based on the load status of the distributed computing nodes.

[0033] After generating multiple initial shards, an independent data transformation task is created for each shard. For example, for the initial shard of electronic products, a data transformation task is created specifically for converting the data format of that shard. Then, the system monitors the load status of the distributed computing nodes in real time. The load status can be measured by metrics such as node CPU utilization and memory utilization. If a computing node has low CPU utilization and a lot of free memory, it indicates that the node is idle. The system will then distribute the data transformation tasks to these idle computing nodes to ensure that the tasks can be executed efficiently and to avoid situations where one node is overloaded while other nodes are idle.

[0034] Step 123: In each distributed computing node, the pre-configured format conversion rule set is invoked to perform structured data pattern alignment and unstructured data semantic standardization on the allocated initial shards, generating intermediate standardized shards.

[0035] On the distributed computing nodes, the initial shards are processed according to a pre-configured set of format conversion rules. For structured data, such as data in a product information table, schema alignment is performed. For example, the data type of the product price field in the source database may be string, while the target database requires a numeric type. The format conversion rule set will guide the nodes to convert the data type of this field to conform to the schema requirements of the target database. For unstructured data, such as text information in product description fields, semantic standardization is performed. A pre-trained model identifies key information in the text and converts it into a unified format. For example, product material information expressed in different ways is uniformly converted into standard terminology, ultimately generating intermediate standardized shards.

[0036] In a preferred embodiment, step 123 includes:

[0037] Step 1231: Perform field mapping matching on the structured data tables in the initial shards. The RPA script automatically compares the source field names with the field definitions in the target database to generate a field mapping relationship table.

[0038] In the initial sharding of the aforementioned database system, the structured data tables contain various fields. The RPA script automatically compares the source field names with the field definitions in the target database. For example, the "product_name" field in the product information table of the source database might correspond to the field definition "product name" in the target database. The RPA script identifies this correspondence and generates a field mapping table. This table records the one-to-one correspondence between source and target fields, providing accurate guidance for subsequent data transformation.

[0039] Step 1232: Perform automated type conversion, including string encoding conversion, date format unification, and numerical precision adjustment, on fields with inconsistent data types according to the field mapping relationship table to obtain the converted structured data table.

[0040] Based on the generated field mapping table, if an inconsistency in the data type of a field is found, automatic type conversion will be performed. For example, if the product listing date field in the source database is in the format "YYYY / MM / DD", while the target database requires the format "YYYY-MM-DD", then the date format of that field will be standardized. If the precision of a numeric field differs between the source and target databases, the numeric precision will also be adjusted. Through these operations, a converted structured data table that meets the requirements of the target database is obtained.

[0041] Step 1233: Perform semantic parsing on the unstructured data blocks in the initial shards, extract key entities and contextual relationships through a pre-trained NLP model, and transform the unstructured data into target structured data blocks containing entity labels and relationship graphs.

[0042] In the initial unstructured data blocks, such as product description text, the pre-trained NLP model comes into play. The model analyzes the text content, extracting key entities, such as the product's brand and model number, as well as the contextual relationships between these entities. For example, if the product description mentions "a certain brand of mobile phone is equipped with a certain model of processor," the model will identify entities such as "brand," "mobile phone," and "processor," and construct a relationship graph between them. Then, this information is organized into target structured data blocks containing entity labels and relationship graphs, facilitating subsequent data processing and storage.

[0043] Step 1234: Reorganize the transformed structured data table and the target structured data block according to the storage rules of the target database, and add shard identifiers and check codes to generate intermediate standardized shards.

[0044] The processed and transformed structured data tables and target structured data blocks are recombined according to the target database's storage rules. For example, if the target database requires data to be stored in a specific order, the relevant data will be arranged accordingly. Simultaneously, to ensure data accuracy and traceability, fragment identifiers and checksums are added to the recombined data. The fragment identifier uniquely identifies which initial fragment the intermediate normalized fragment originated from, and the checksum is used to detect whether errors occurred during data transmission or storage, ultimately generating intermediate normalized fragments.

[0045] Step 124: Aggregate all intermediate standardized fragments to obtain fragment aggregation results, perform data integrity verification and deduplication on the fragment aggregation results, and generate the target data set.

[0046] After intermediate standardized shards are generated on each distributed computing node, these shards are aggregated to form a unified whole, resulting in a shard aggregation result. This aggregation result is then subjected to data integrity verification to check for missing or corrupted data. For example, it checks the completeness of product information and whether any key information for a particular product is missing. Simultaneously, deduplication is performed to remove duplicate data records. For instance, if two identical product records exist, only one is retained. Through these operations, the target dataset is finally generated, a dataset that has undergone rigorous processing to meet the requirements of the target database.

[0047] Step 130: Detect the task execution status of the distributed computing node. When the completion event of the generation of the target data set is detected, activate the data synchronization channel and transmit the target data set to the target database.

[0048] In the aforementioned database system, it is necessary to monitor the task execution status of the distributed computing nodes in real time. The system continuously checks whether the data transformation tasks of each node have been completed. When all nodes have completed their data transformation tasks and generated the target dataset, a target dataset generation completion event is triggered. At this time, the system activates the data synchronization channel, which is responsible for securely and accurately transmitting the target dataset to the target database, ensuring that the data is updated to the target database in a timely manner.

[0049] As another optional embodiment, step 130 includes:

[0050] Step 131: Use the RPA task status collector to obtain the execution status parameters of each data shard task in the distributed computing node in real time. The execution status parameters include at least the task progress percentage and the exception error code.

[0051] In the distributed computing environment of the aforementioned database system, the RPA task status collector operates continuously, communicating with each distributed computing node to obtain the execution status parameters of each data shard task in real time. For example, for a data shard task, the collector will obtain its task progress percentage, such as showing 70% completion, and will also obtain exception error codes; if no errors occur, the error code will be empty. Through these parameters, the system can comprehensively understand the execution status of each task and promptly identify potential problems.

[0052] Step 132: When it is detected that the progress percentage of all data sharding tasks has reached the preset percentage and the error code is empty, generate a completion event for the target data set and activate the data synchronization channel running command.

[0053] In this embodiment, the system sets a preset percentage, such as 95%. When the RPA task status collector reports that the progress percentage of all data sharding tasks has reached this preset percentage, and all task error codes are empty, it means that all data transformation tasks have been successfully completed and the target data set has been generated. At this time, the system generates a target data set generation completion event and activates the data synchronization channel operation command, notifying relevant components to prepare to open the data synchronization channel for data transmission.

[0054] Step 133: According to the data synchronization channel operation instruction, call the pre-configured channel connector to establish a bidirectional encrypted channel with the target database. The channel connector loads the corresponding communication protocol driver based on the type of the target database.

[0055] It's understandable that upon receiving the data synchronization channel operation command, the system will invoke the pre-configured channel connector. Different types of target databases may require different communication protocol drivers for connection. For example, if the target database is a relational database, the channel connector will load a communication protocol driver suitable for that database. Through this communication protocol driver, a bidirectional encrypted channel is established with the target database, ensuring data security and integrity during data transmission and preventing data theft or tampering.

[0056] Step 134: Arrange the target data set in the distributed storage nodes according to the fragment identifier order for transmission sequence, and inject a flow control strategy to optimize network congestion.

[0057] In the aforementioned database system, distributed storage nodes store the target data set. To ensure orderly and efficient data transmission to the target database, the target data set is sequenced according to the order of its fragment identifiers. For example, data fragments from different product categories are arranged in a specific order. Simultaneously, to prevent network congestion from affecting data transmission speed and quality, flow control strategies are implemented. For instance, the transmission rate of each data fragment is dynamically adjusted based on real-time network bandwidth conditions to ensure efficient use of network resources.

[0058] In another preferred embodiment, step 134 includes:

[0059] Step 1341: Parse the fragment identifiers of each data fragment in the target data set, extract the global sequence number field and the fragment generation timestamp, generate the initial transmission sequence in ascending order of the global sequence number field value, and calculate the waiting transmission time of each data fragment based on the difference between the fragment generation timestamp and the current system timestamp.

[0060] In the target dataset, each data shard has its own shard identifier. The system parses these identifiers, extracting the global sequence number field and the shard generation timestamp. For example, a shard identifier might contain "003_2023-10-05 11:00:00", where "003" is the global sequence number field and "2023-10-05 11:00:00" is the shard generation timestamp. The data shards are sorted in ascending order of their global sequence number values ​​to generate an initial transmission sequence. Then, the current system timestamp is subtracted from the shard generation timestamp to calculate the waiting time for each data shard, providing a basis for subsequent priority adjustments.

[0061] Step 1342: When the waiting transmission time exceeds the preset transmission time threshold, the priority weight of the corresponding data fragment is increased to the first preset level, and the real-time network parameter set between the distributed storage node and the target database is collected simultaneously. The network parameter set includes the normalized bandwidth utilization coefficient, latency attenuation coefficient and packet loss penalty coefficient.

[0062] If the waiting time for a data fragment to be transmitted exceeds a preset transmission timeliness threshold, for example, if the preset threshold is 10 minutes and a fragment's waiting time reaches 15 minutes, then the priority weight of that data fragment will be raised to the first preset level. Simultaneously, the system will collect a set of real-time network parameters between the distributed storage nodes and the target database. These parameters, after normalization, can more accurately reflect the network status. For example, the bandwidth utilization coefficient can represent the current network bandwidth usage, the latency attenuation coefficient reflects the degree of data transmission latency, and the packet loss penalty coefficient measures the impact of packet loss on transmission.

[0063] Step 1343: The priority weights and normalized network parameters are weighted and calculated to generate a comprehensive transmission priority score for each data fragment. The initial transmission sequence is reordered based on the score to generate an optimized transmission sequence. The transmission window size of each data fragment in the transmission sequence is adjusted according to the ratio of the current available bandwidth to the data fragment size.

[0064] Optionally, the enhanced priority weights are weighted together with the normalized network parameters. For example, a certain weight value is assigned to the priority weights, and corresponding weight values ​​are also assigned to the bandwidth utilization coefficient, latency attenuation coefficient, and packet loss penalty coefficient. The comprehensive transmission priority score of each data fragment is obtained through weighted calculation. Based on this score, the initial transmission sequence is reordered, with higher-priority data fragments placed first. At the same time, the transmission window size of each data fragment is reasonably adjusted according to the ratio of the currently available bandwidth to the data fragment size. For example, if a data fragment is large and the currently available bandwidth is limited, its transmission window size is appropriately reduced to ensure that the network does not become congested due to the transmission of a large fragment.

[0065] Step 1344: Encapsulate the data in the transmission sequence into data packets carrying fragment identifiers and checksums using a preset flow control strategy, start the parallel transmission channel, and monitor the transmission delay and packet loss rate changes of each data packet in real time during the transmission process.

[0066] Optionally, data in the transmission sequence is fragmented and encapsulated according to a preset flow control strategy. Each data packet carries a fragment identifier and a checksum, allowing for easy identification and verification of data during transmission. A parallel transmission channel is activated, enabling multiple data packets to be transmitted simultaneously, improving transmission efficiency. During transmission, the system monitors the transmission latency and packet loss rate of each data packet in real time. For example, network monitoring tools can be used to obtain the time required for each data packet to be transmitted from the distributed storage node to the target database, as well as whether any data packets are lost.

[0067] Step 1345: When the packet loss rate change of any data segment exceeds the preset tolerance threshold, the transmission window size of the data segment is reduced to the second preset level according to the flow control strategy, and the transmission window size of the adjacent data segments is increased synchronously based on the current overall bandwidth utilization rate to maintain the balance of bandwidth utilization.

[0068] Understandably, if the packet loss rate change of a data segment exceeds a preset tolerance threshold (e.g., the preset tolerance threshold is 5%, and the packet loss rate change of that data segment reaches 8%), then according to the flow control strategy, the transmission window size of that data segment will be reduced to the second preset level. Simultaneously, to maintain a balance in overall bandwidth utilization, the system will synchronously increase the transmission window size of adjacent data segments based on the current overall bandwidth utilization. For example, if the current overall bandwidth utilization is 70%, when reducing the transmission window size of the data segment with the excessive packet loss rate, the transmission window size of its adjacent data segments will be increased accordingly, keeping the overall bandwidth utilization within a reasonable range and ensuring the stability and efficiency of data transmission.

[0069] Step 1346: After the target database returns a write confirmation signal carrying a fragment identifier, update the transmission log status of the corresponding fragment identifier; if the write confirmation signal of the target fragment identifier is not received within the preset timeout period, extract the corresponding data fragment from the distributed storage node based on the flow control strategy, repackage it, insert it into the head of the transmission sequence, and perform retransmission.

[0070] Once the target database successfully receives a data shard and completes the write operation, it returns a write acknowledgment signal carrying the shard identifier. Upon receiving this signal, the system immediately updates the transmission log status of the corresponding shard identifier, marking it as successfully transmitted and written. For example, it records detailed information such as the transmission time and write time corresponding to the shard identifier in the transmission log.

[0071] However, if the system does not receive a write acknowledgment signal for the target fragment identifier within the preset timeout period (e.g., a preset timeout of 30 seconds), it indicates a potential problem with the transmission of that data fragment. In this case, based on flow control strategies, the system will retrieve the corresponding data fragment from the distributed storage node, repackage it, and ensure data integrity and accuracy. Then, the repackaged data fragment is inserted at the head of the transmission sequence, prioritizing retransmission to ensure successful transmission to the target database.

[0072] Step 135: During transmission, a target verification mechanism is used to perform cyclic redundancy check on the data packets of each fragment. If the check fails, the retransmission process of the corresponding fragment is executed until all fragments are completely written into the buffer queue of the target database.

[0073] During the data transmission from distributed storage nodes to the target database, a target verification mechanism is used to perform cyclic redundancy check on each data packet to ensure data accuracy. Cyclic redundancy check is a method that detects data integrity by calculating the checksum of data packets. For example, for each data packet, the system calculates a checksum value according to a commonly used algorithm and appends it to the data packet. When the target database receives the data packet, it recalculates the checksum value and compares it with the received checksum value.

[0074] If the comparison results in a verification failure, meaning the two checksum values ​​are not equal, it indicates that an error may have occurred in the data packet during transmission. In this case, the system will execute the retransmission process for the corresponding fragment. As mentioned earlier, the data fragment is retrieved again from the distributed storage node, repackaged, and transmitted again. This process will be repeated continuously until all fragments pass verification and are completely written into the target database's buffer queue. This ensures that the data transmitted to the target database is accurate and provides a reliable foundation for subsequent data processing and use.

[0075] Step 140: Detect and identify conflicts based on the RPA conflict handling strategy configured in the target database. When a primary key conflict or unique constraint conflict is detected, update the target database based on the preset conflict merging indication information and generate a data synchronization log.

[0076] In the aforementioned database system, the target database has been configured with an RPA conflict handling strategy. When the target dataset is transmitted to the target database, the system will perform conflict detection and identification based on this strategy. For example, when inserting new product data, a primary key conflict may occur, meaning the primary key of the newly inserted product is the same as the primary key of an existing product in the target database; or a unique constraint conflict may occur, such as a unique identifier field for a product already having the same value in the database.

[0077] Once these conflicts are identified, the system will update the target database according to the preset conflict merging instructions. Simultaneously, to record the conflicts throughout the data synchronization process, a data synchronization log will be generated. This log helps administrators understand the problems that occur during data synchronization, facilitating subsequent analysis and processing.

[0078] In an alternative embodiment, step 140 includes:

[0079] Step 141: Configure a conflict detection rule set in the target database. The conflict detection rule set includes primary key duplicate detection rules, foreign key constraint verification rules, and unique index conflict rules.

[0080] In the target database of the aforementioned database system, the administrator will pre-configure a set of conflict detection rules. Primary key duplication detection rules are used to check whether the primary key of a newly inserted or updated data record is duplicated with an existing primary key in the database. For example, in a product information table, each product has a unique product ID as its primary key; this rule can be used to detect whether the ID of a newly inserted product already exists.

[0081] Foreign key constraint validation rules are used to ensure the correctness of foreign key relationships. For example, the product ID in the order table is a foreign key that relates to the primary key in the product information table. This rule will verify whether the product ID referenced in the order table exists in the product information table to ensure data consistency.

[0082] In addition, unique index conflict rules are used to detect duplicate values ​​in fields that have unique indexes. For example, if a product's barcode field has a unique index, this rule can prevent the insertion of different product records with the same barcode. These rules comprehensively perform conflict detection on the data in the target database.

[0083] Step 142: Create a data write transaction in the temporary storage area of ​​the target database, and sequentially apply the conflict detection rule set to perform conflict scanning on the target data set to generate a list of conflicting data.

[0084] Once the target dataset arrives at the target database, the system creates a data write transaction in the target database's temporary storage area and performs a series of operations on the target dataset through this transaction. Then, the system sequentially applies the rules from the conflict detection rule set to perform a comprehensive conflict scan of the target dataset.

[0085] For example, first, a primary key duplication check rule is applied to check whether the primary key of each data record in the target dataset is a duplicate of an existing primary key in the target database. Next, a foreign key constraint validation rule is applied to check whether foreign key relationships are correct. Finally, a unique index conflict check rule is applied to check whether there are conflicts in fields with unique indexes. During the scan, all detected conflicting data is recorded, generating a conflict data list. This list details relevant information for each conflicting data, such as which table the conflict occurred in, the fields involved, and the conflict type, providing a basis for subsequent conflict resolution.

[0086] Step 143: Match the preset conflict merging instruction information according to the conflict type in the conflict data list. The conflict merging instruction information includes timestamp overwrite strategy, field priority strategy and manual review strategy. The conflict type includes primary key conflict or unique constraint conflict.

[0087] After generating the list of conflicting data, the system matches the pre-defined conflict merging instructions based on the conflict types recorded in the list. For primary key conflicts, if a timestamp overwrite strategy is used, it means comparing the source transaction timestamp of the conflicting data record with the timestamp of the current record in the target database, and using the record corresponding to the newer timestamp for overwriting. For example, if the source transaction timestamp of a newly inserted product record is newer than the timestamp of the same primary key product record in the target database, then the new record will overwrite the old record.

[0088] For uniqueness constraint conflicts, if a field priority strategy is used, a pre-configured field priority mapping table will be used to determine which fields among the conflicting fields have higher weights. For example, if a conflict occurs in a unique identifier field of a product, according to the field priority mapping table, the product name field may have a higher weight. In this case, the data of the product name field will be retained, and other related fields will be adjusted.

[0089] In the event of complex or special conflict situations, a manual review strategy may be adopted, whereby conflict data records are submitted to the administrator for manual judgment and processing to ensure that the conflict is resolved reasonably.

[0090] Step 144: The RPA robot executes the automated operation corresponding to the conflict merging instruction information, updates the conflict data records in the target database, and writes the update result to the data synchronization log.

[0091] Understandably, once the conflict merging instructions are determined, the RPA robot will execute the corresponding automated operations. For example, if a timestamp overwrite strategy is used, the RPA robot will, according to the rules, overwrite the old records in the target database with the new records, thus resolving the primary key conflict. For a field priority strategy, the RPA robot will adjust and merge the data in the conflicting fields based on the field priority mapping table, updating the conflicting data records in the target database.

[0092] After completing these update operations, the RPA robot will write the update results in detail to the data synchronization log. The log will record the update time, the tables and fields involved, the operations performed, and a comparison of the data before and after the update. As a result, administrators can clearly understand the handling of conflicts during the data synchronization process by viewing the data synchronization log, which facilitates auditing and troubleshooting.

[0093] In an optional embodiment, step 144 includes:

[0094] Step 1441: Based on the conflict type in the conflict data list, extract the source transaction timestamp and the current record timestamp of the target database from the conflict data record, perform time unit standardization conversion based on the source transaction timestamp and the current record timestamp, and generate time coverage priority parameters.

[0095] Understandably, for each conflicting data record in the conflict list, the system will first determine the record that needs to be processed based on the conflict type. If it is a primary key conflict or a unique constraint conflict involving timestamps, the system will extract the source transaction timestamp and the timestamp of the current record in the target database from the conflicting data record.

[0096] For example, consider a primary key conflict record for product information. The source transaction timestamp is "2023-10-06 14:30:00," while the timestamp of the current record in the target database is "2023-10-05 11:45:00." To accurately compare these two timestamps, a time unit standardization conversion is needed. This converts the two timestamps into a unified time representation, such as the number of seconds since a fixed start time. This standardization conversion generates a time coverage priority parameter, which clearly indicates which timestamp is "newer," providing a quantitative basis for subsequent timestamp coverage strategies.

[0097] Step 1442: Combine the pre-configured field priority mapping table, extract the normalized weight values ​​of the conflicting fields in the conflicting data records and generate field priority parameters. Then, weight and merge the time coverage priority parameters and field priority parameters according to a preset ratio to generate a comprehensive priority sorting sequence for each conflicting data record.

[0098] When processing conflicting data records, if a field priority strategy is involved, the system will use a pre-configured field priority mapping table. For example, for conflicting data records in the product information table, conflicting fields might include product name, price, and inventory. The field priority mapping table determines the normalized weight value for each conflicting field. For instance, the normalized weight value for product name is 0.6, for price it is 0.3, and for inventory it is 0.1.

[0099] These normalized weight values ​​are used as field priority parameters and weighted together with the previously generated time coverage priority parameters according to a preset ratio. For example, the preset weight of the time coverage priority parameter is 0.4, and the weight of the field priority parameter is 0.6. Through weighted calculation, a comprehensive priority ranking sequence for each conflicting data record is generated. This sequence helps the system determine the processing priority of each conflicting data record, so as to handle conflicts more rationally.

[0100] Step 1443: Traverse conflicting data records based on the comprehensive priority sorting sequence: If there is a primary key conflict and the comprehensive priority exceeds the preset priority, generate a target record replacement script and perform an overwrite update; if there is a uniqueness constraint conflict and the priority does not reach the preset priority, generate a conflict field merging script based on field priority comparison and retain the weight field data.

[0101] After generating the overall priority sorting sequence, the system iterates through the conflicting data records according to this sequence. For primary key conflicts, if the overall priority of a conflicting data record exceeds the preset priority (e.g., the preset priority is 0.5, and the record's overall priority is 0.6), this record needs to be processed first and should be updated by overwriting. The system generates a target record replacement script, which is executed by an RPA robot to overwrite the old record in the target database with the new record.

[0102] For uniqueness constraint conflicts, if the priority of a conflicting data record does not reach the preset priority, the system will generate a conflict field merging script based on field priority comparison. For example, when a conflict occurs in a unique identifier field of a product, the script will determine the field with higher weight according to the field priority mapping table. If the product name field has a higher weight, the script will retain the data of the product name field and adjust and merge other related fields to resolve the uniqueness constraint conflict.

[0103] Step 1444: After the script is executed, the RPA robot sends a verification request containing a transaction identifier to the target database. The returned actual data is compared with the expected value in the script field by field. If there is a difference, the difference record is re-inserted into the conflict data list and an alarm event is activated.

[0104] After the RPA robot executes the target record replacement script or conflict field merging script, to ensure the accuracy of the update operation, it will send a verification request containing a transaction identifier to the target database. The target database will then return the updated actual data.

[0105] The system will compare the returned actual data with the script's expected values ​​field by field. For example, when updating product information, it will compare the actual and expected values ​​of each field, such as product name, price, and inventory, to see if they match. If a discrepancy is found, such as the actual price not matching the expected price, the discrepancy record will be re-inserted into the conflict data list for further analysis and processing. Simultaneously, an alarm event will be activated to notify the administrator of the data inconsistency, requiring timely attention and resolution.

[0106] Step 1445: Associate the execution results of the target record replacement script or conflict field merging script, the field-by-field comparison results, and the transaction identifier to generate a log entry with a version identifier; synchronously write the log entry into the data synchronization log partition of the distributed storage node and the audit table of the target database, and generate an inverse index relationship with the original data shard.

[0107] Optionally, the system will associate the execution results of the target record replacement script or conflict field merging script, the field-by-field comparison results, and the transaction identifier. For example, the updated product information record after the script is executed, the differences found during the comparison process, and the transaction identifier corresponding to the operation are combined to generate a log entry with a version identifier. This version identifier can be used to distinguish different versions of operation records, facilitating traceability and management.

[0108] Then, the log entry is synchronously written to the data synchronization log partition of the distributed storage node and the audit table of the target database. The data synchronization log partition records detailed information throughout the data synchronization process, facilitating querying and analysis by administrators. The audit table of the target database can be used for auditing and compliance checks. Simultaneously, to facilitate tracing from log records to the original data shards, an inverse index relationship with the original data shards is generated. For example, by recording the identifier of the original data shard in the log entry, a connection is established from the log to the original data, enabling administrators to quickly locate and view the relevant original data when needed.

[0109] In another embodiment, the method further includes steps 210, 220, 230, or 240.

[0110] Step 210: Monitor the running status of distributed computing nodes in real time during data synchronization. When a node failure or task timeout is detected, automatically reassign the unfinished tasks to healthy nodes.

[0111] During the data synchronization process of the aforementioned database system, the system monitors the operational status of distributed computing nodes in real time. It determines whether a node is functioning correctly by monitoring metrics such as CPU utilization, memory utilization, and network connectivity. For example, if a computing node's CPU utilization consistently exceeds 90%, its memory utilization is close to 100%, and it experiences unstable network connectivity, this may indicate that the node is about to fail.

[0112] Alternatively, if a data sharding task fails to complete within a preset time (i.e., the task times out, for example, a task with a preset completion time of 60 minutes that is still incomplete after 90 minutes), the system will automatically reassign the incomplete task from the problematic node to a healthy node. For instance, an incomplete data sharding task can be moved from the faulty node to a healthy node with low CPU and memory usage and a stable network connection to continue execution, ensuring that the data synchronization task can be completed smoothly without interruption due to a problem with a single node.

[0113] Step 220: If a format parsing error occurs during the data conversion process, the RPA robot calls the backup conversion rule set to retry, and after the retry fails, the erroneous data fragment is extracted and sent to the repair queue.

[0114] During the format conversion process of the initial data shards on distributed computing nodes, format parsing errors may occur. For example, when converting unstructured data into target structured data blocks, the pre-trained NLP model may fail to correctly parse certain text content, leading to format conversion failure.

[0115] When this occurs, the RPA bot will invoke a backup transformation rule set to retry. The backup rule set may employ a different algorithm or model for format conversion, hoping to successfully process the data. For example, if the original transformation rule set used a machine learning-based approach, the backup rule set might use a rule engine-based approach.

[0116] If the retry still fails, the system will extract the error data fragment. For example, for product description text that cannot be correctly converted, this text will be extracted as an error data fragment. This error data fragment will then be sent to the repair queue. The repair queue is used to process error data; administrators or subsequent processes can retrieve error data from this queue for further analysis and repair.

[0117] Step 230: When the connection is interrupted during the writing process of the target database, the breakpoint position is automatically recorded and the writing operation is resumed from the breakpoint position after the connection is restored.

[0118] During the process of writing the target dataset into the target database, connection interruptions may occur due to network failures, database server malfunctions, or other reasons. For example, when writing product information data, a sudden network outage may cause some data to fail to be written successfully.

[0119] When this happens, the system automatically records the breakpoint location. For example, it records the position of the last piece of data that has been successfully written, as well as information about the data that should be written next. This breakpoint location can be represented by the data offset, record number, etc.

[0120] Once the connection is restored, the system will resume the write operation from the breakpoint. Based on the previously recorded breakpoint information, it will locate the unwritten data and continue writing it to the target database. For example, if 100 product information records have already been successfully written, and the breakpoint is recorded as the 100th record, after the connection is restored, the system will start writing from the 101st record, ensuring data integrity and continuity.

[0121] Step 240: Periodically analyze error patterns in the data synchronization log, and adjust the data sharding task allocation strategy and conflict detection rule set to optimize synchronization efficiency.

[0122] The system periodically analyzes the data synchronization logs, for example, once every morning. The data synchronization logs record various error messages, such as node failures, format parsing errors, and conflict resolution failures. By analyzing these error messages, error patterns are extracted.

[0123] For example, if you discover frequent format parsing errors due to issues with certain field formats within a specific time period, or if a data sharding task frequently times out on certain nodes, adjust the data sharding task allocation strategy based on these error patterns. If a data sharding task is prone to problems on certain nodes, adjust the allocation strategy to avoid assigning the task to those nodes.

[0124] Simultaneously, the conflict detection rule set is adjusted based on error patterns. For example, if a certain type of conflict is found to occur frequently and be processed inefficiently, the rules for that type of conflict in the conflict detection rule set are optimized. This could involve adding more granular pre-detection conditions or adjusting parameters in the conflict merging indication information, enabling conflicts to be detected and processed more quickly and accurately. This optimizes the overall efficiency of data synchronization, reduces errors, and improves the stability and reliability of data synchronization.

[0125] As an independently implementable technical solution, after updating the target database based on the preset conflict merging indication information and generating a data synchronization log, the solution further includes: periodically collecting the conflict type distribution frequency and task execution time interval in the data synchronization log; converting the conflict type distribution frequency into a conflict weight vector according to a preset normalization rule; calculating the difference between the task execution time interval and a preset reference time parameter to generate a time deviation coefficient; inputting the conflict weight vector and the time deviation coefficient into a pre-trained synchronization strategy optimization model; outputting a sharding rule adjustment coefficient and a conflict detection sensitivity parameter; updating the sharding size threshold of the data sharding task based on the sharding rule adjustment coefficient; adjusting the priority weight of the timestamp overlay strategy in the RPA conflict handling strategy according to the conflict detection sensitivity parameter; generating the data sharding task allocation instruction and conflict merging rule sequence for the next cycle based on the updated sharding size threshold and priority weight; and injecting them into the RPA robot configuration files of the distributed computing node and the target database.

[0126] During the data synchronization process of the aforementioned database system, in order to continuously optimize the synchronization strategy, the system periodically performs in-depth analysis of the data synchronization logs. For example, data collection is conducted weekly. From the data synchronization logs, two key pieces of information are carefully collected: the frequency of conflict type distribution and the task execution time interval.

[0127] For conflict type distribution frequency, the number of times each type of conflict (such as primary key conflict, uniqueness constraint conflict, etc.) occurs within a certain period is recorded. For example, in the past week, primary key conflicts occurred 20 times, and uniqueness constraint conflicts occurred 15 times. Then, these conflict type distribution frequencies are converted into a conflict weight vector according to a preset normalization rule. The normalization rule can be to divide the frequency of each conflict type by the total number of conflicts to obtain the relative weight of each conflict type. For example, if the total number of conflicts is 35, then the weight of primary key conflict is 20 / 35, the weight of uniqueness constraint conflict is 15 / 35, and so on, forming a vector that clearly reflects the relative importance of various conflict types in the data synchronization process.

[0128] Simultaneously, task execution time interval information is collected. The time taken for each data shard task from start to finish is calculated and compared with a preset baseline time parameter. For example, if the preset baseline time parameter is 30 minutes and the actual execution time of a data shard task is 40 minutes, a time deviation coefficient is generated by calculating the difference. An exemplary time deviation coefficient is 40-30=10 minutes (those skilled in the art can consider various additional factors according to actual needs).

[0129] The generated conflict weight vector and time deviation coefficient are used as inputs to a pre-trained synchronization strategy optimization model. This model, trained on a large amount of data, can intelligently analyze and process the input information. The model outputs the partitioning rule adjustment coefficients and conflict detection sensitivity parameters.

[0130] Based on the sharding rule adjustment coefficient, the shard size threshold for the data sharding task is updated. For example, if the sharding rule adjustment coefficient indicates that the current shard size may lead to low data processing efficiency and the shard size needs to be appropriately increased, then the shard size threshold is increased accordingly. For instance, if the original shard size threshold was based on 1000 records per shard, the threshold might be adjusted to 1500 records per shard based on the adjustment coefficient.

[0131] Based on the conflict detection sensitivity parameters, adjust the priority weight of the timestamp overlay strategy in the RPA conflict handling strategy. If the conflict detection sensitivity parameters indicate a greater need for timestamp accuracy, then increase the priority weight of the timestamp overlay strategy within the overall conflict handling strategy. For example, if the original priority weight of the timestamp overlay strategy was 0.4, adjust it to 0.6 based on the parameters.

[0132] Finally, based on the updated shard size threshold and priority weights, the data sharding task allocation instructions and conflict merging rule sequence for the next cycle are generated. These instructions and rule sequences are injected into the RPA robot configuration files of the distributed computing nodes and the target database. Thus, during the data synchronization process of the next cycle, the distributed computing nodes and the RPA robot will operate according to the new strategies and rules, thereby continuously optimizing the efficiency and accuracy of data synchronization.

[0133] As an independently implementable technical solution, after updating the target database based on the preset conflict merging indication information and generating a data synchronization log, the solution further includes: extracting data blocks from the target data set already written to the target database according to shard identifiers, calculating the hash digest value of each data block and generating a shard hash set; obtaining the corresponding initial data set from the distributed storage node, and generating a source shard hash set according to the same shard identifier and hash algorithm; comparing the shard hash set with the source shard hash set item by item, and if there are shard identifiers with inconsistent hash digest values, generating a difference shard identifier list and performing a data repair task; extracting the original data shards from the distributed storage node according to the difference shard identifier list, re-performing the format conversion operation to generate repaired data shards, and writing the repaired data shards into the independent repair storage area of ​​the target database through the data synchronization channel based on the RPA conflict handling strategy; after the repaired data shards are written, updating the repair status flag in the data synchronization log, and writing the mapping relationship between the difference shard identifier list and the physical address of the repair storage area into an inverted index table.

[0134] After data synchronization is complete, to ensure the accuracy and consistency of data in the target database, an integrity check is performed on the target data set already written to the target database. First, data blocks are extracted from the target database according to shard identifiers. For example, for product information data, data from different shards is extracted separately based on the previously defined shard identifiers.

[0135] Then, for each extracted data block, its hash digest value is calculated. The hash algorithm can be a widely used algorithm, such as SHA-256. By hashing the data block, a unique hash digest value is generated, which can represent the content characteristics of the data block. The hash digest values ​​of all data blocks are collected to generate a fragmented hash set.

[0136] Simultaneously, the corresponding initial data set is obtained from the distributed storage nodes. Using the same shard identifier and hash algorithm, each shard data block in the initial data set is hashed to generate a source shard hash set.

[0137] Next, the shard hash set is compared item by item with the source shard hash set. The hash digest values ​​of data blocks corresponding to the same shard identifier are compared one by one in both sets. If shard identifiers with inconsistent hash digest values ​​are found, it indicates that a problem may have occurred during the synchronization process for that shard. For example, if the hash digest value of a data block in the target database corresponding to a certain shard identifier differs from the hash digest value of the original data block in the distributed storage node, that shard identifier is recorded, generating a list of differing shard identifiers.

[0138] Once a list of differential shard identifiers is generated, a data repair task is executed. Based on this list, the original data shards are retrieved from the distributed storage nodes. For example, based on a specific shard identifier in the differential shard identifier list, the corresponding original data shard is accurately located from the distributed storage nodes.

[0139] Then, the format conversion operation is re-performed on the extracted raw data fragments. Following the previous data conversion process, the fragmented data is subjected to structured data schema alignment and unstructured data semantic standardization to generate repaired data fragments.

[0140] Based on the RPA conflict handling strategy, repair data shards are written to the independent repair storage area of ​​the target database through a data synchronization channel. During the writing process, the data synchronization process and rules are followed to ensure that the data is written accurately.

[0141] Once the repaired data shard has been written, the repair status flag in the data synchronization log is updated. The log explicitly records that the data shard has been repaired and successfully written, facilitating subsequent queries and auditing. Simultaneously, the mapping relationship between the list of differential shard identifiers and the physical address of the repair storage area is written to an inverted index table. This inverted index table helps administrators quickly locate and find the position of the problematic data shard in the repair storage area, enabling further inspection and processing when needed. Through this series of operations, the data in the target database is ensured to remain accurate and complete, improving data reliability and availability.

[0142] In practical applications, technical details and processes can be supplemented and optimized based on existing technologies such as database log parsing, distributed computing frameworks, and NLP model application experience.

[0143] For example, to address the insufficient public availability of data extraction scripts generated by RPA robots, existing dynamic SQL generation technologies can be used to automatically match operation parameters with script templates. This, combined with an operation type mapping rule base (such as MyBatis's dynamic SQL mechanism), enables the assembly of conditional statements, thus ensuring the feasibility of script generation. For NLP semantic parsing of unstructured data, the entity recognition interface of existing open-source pre-trained models (such as BERT) can be introduced. By configuring entity extraction rules and constructing templates using relational graphs, key entities and contextual relationships in the text can be transformed into structured fields.

[0144] Regarding the quantification of sharding rules, existing distributed database sharding strategies (such as Cassandra's consistent hashing algorithm) can be used. Data table association graph analysis tools can be employed to calculate the frequency of connections between tables, and field dependency detection algorithms (such as functional dependency derivation) can be combined to dynamically generate sharding weight coefficients, thereby achieving a quantitative assessment of data distribution characteristics. For parameter calculations in flow control strategies, the sliding window mechanism of TCP congestion control algorithms can be referenced. Existing network quality probe tools can be used to obtain parameters such as bandwidth, latency, and packet loss rate in real time, and these parameters can be normalized using an exponentially weighted moving average algorithm to establish a dynamic priority adjustment model.

[0145] In the timestamp standardization process, ISO8601 time format conversion technology can be used, combined with a time zone offset compensation algorithm, to achieve a unified time base across systems. For field priority mapping in conflict detection, a field impact assessment model can be built using existing data lineage analysis tools, and field weight coefficients can be configured using a business rule engine to ensure the configurability of the conflict merging strategy.

[0146] This design enables the complete process from data change monitoring and distributed processing to conflict resolution.

[0147] This application embodiment utilizes an RPA robot deployed on the source database server to monitor data changes, enabling real-time capture of changes in both structured and unstructured data and timely generation of an initial data set, ensuring data timeliness and integrity. The initial data set is sharded and distributed to distributed computing nodes for parallel format conversion, significantly improving data processing efficiency and fully leveraging the advantages of distributed computing to quickly generate the target data set. By real-time monitoring of the task execution status of the distributed computing nodes, a data synchronization channel is activated once the target data set is generated, ensuring accurate and efficient data transmission to the target database and guaranteeing timely data flow. Conflict detection and identification are performed in the target database based on the configured RPA conflict handling strategy, and the database is updated and synchronization logs are generated according to preset information. This effectively resolves conflicts during data synchronization, ensuring data consistency and accuracy, improving the reliability and stability of the entire data synchronization process, and providing solid data support for the stable operation of the RPA business system.

[0148] In summary, the embodiments of this application effectively improve the timeliness, efficiency, and consistency of data processing by using RPA robots to monitor data changes in real time, employing distributed parallel processing, and implementing a precise conflict detection and handling mechanism.

[0149] It is important to note that, regarding the data acquisition and processing procedures involved in this application's embodiments, all data operations are performed with the user's full knowledge and explicit authorization. During user registration and service usage, the system informs the user of the data collection scope, purpose of use, and sharing scenarios through an explicit privacy agreement, and obtains the user's specific authorization for the collection, processing, and synchronization of relevant data such as product information and order records through a checkbox confirmation method. For operations involving sensitive user personal data (such as order address modification), an enhanced authorization mechanism is adopted, confirming the user's intention through secondary verification. During data synchronization, the RPA robot only processes non-identity-related data that has been anonymized, and key fields are protected end-to-end using the AES-256 encryption algorithm. Furthermore, all data change event monitoring is restricted to execution within the security boundaries of the enterprise's internal authorized server. The system regularly conducts data security impact assessments, controls data access levels through the principle of least privilege, and audit logs fully record the data operation trajectory for compliance review. Users can view the authorization scope, revoke data sharing licenses, or initiate data deletion requests in real time through the enterprise service portal. The system will complete the authorization status synchronization and data residue cleanup within several working days, effectively protecting the user's right to know and right to self-control over data.

[0150] See Figure 2 As shown in the figure, this is a schematic diagram of the basic structure of a database multi-source heterogeneous data synchronization device 200 provided in an embodiment of this application. The database multi-source heterogeneous data synchronization device 200 includes:

[0151] Processor 201;

[0152] Storage device 202, on which computer program 2020 is stored;

[0153] When the computer program 2020 is executed by the processor 201, the processor 201 implements any of the aforementioned database multi-source heterogeneous data synchronization methods.

[0154] Based on the above, a readable storage medium is provided, on which a program or instructions are stored, and when the program or instructions are executed by a processor, the steps of the above method are implemented.

[0155] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

Claims

1. A method for synchronizing heterogeneous data from multiple database sources, characterized in that, include: By monitoring data change events through an RPA robot deployed on the server where the source database is located, an initial data set is generated when a change operation on structured or unstructured data is detected. The initial dataset is sharded to generate multiple data sharding tasks, which are then distributed to distributed computing nodes to perform parallel format conversion operations to generate the target dataset. Based on the data distribution characteristics of the initial data set, the initial data set is divided into multiple initial shards using preset sharding rules, which include data table association, data volume balance, and field dependency relationships. Create an independent data transformation task for each initial shard, and allocate the data transformation task to idle computing nodes based on the load status of the distributed computing nodes; In each distributed computing node, a pre-configured set of format conversion rules is invoked to perform structured data pattern alignment and unstructured data semantic standardization on the allocated initial shards, generating intermediate standardized shards. Field mapping matching is performed on the structured data tables in the initial shards, and the source field names are automatically compared with the field definitions in the target database through RPA scripts to generate a field mapping relationship table. Based on the field mapping relationship table, automated type conversion, including string encoding conversion, date format unification, and numerical precision adjustment, is performed on fields with inconsistent data types to obtain the converted structured data tables. Semantic parsing is performed on the unstructured data blocks in the initial shards. Key entities and contextual relationships are extracted using a pre-trained NLP model, and the unstructured data is transformed into target structured data blocks containing entity labels and relationship graphs. The transformed structured data table and the target structured data block are reorganized according to the storage rules of the target database, and fragmentation identifiers and check codes are added to generate intermediate standardized fragments. All intermediate standardized fragments are aggregated to obtain fragmented aggregation results. Data integrity verification and deduplication are performed on the fragmented aggregation results to generate the target data set. The task execution status of the distributed computing node is detected. When the completion event of the generation of the target data set is detected, the data synchronization channel is activated and the target data set is transmitted to the target database. Conflict detection and identification are performed based on the RPA conflict handling strategy configured in the target database. When a primary key conflict or unique constraint conflict is detected, the target database is updated based on the preset conflict merging indication information, and a data synchronization log is generated.

2. The database multi-source heterogeneous data synchronization method according to claim 1, characterized in that, The process involves using an RPA robot deployed on the server hosting the source database to monitor data change events. When a change operation on structured or unstructured data is detected, an initial data set is generated, including: The RPA robot performs real-time analysis of the operation logs of the source database to extract a set of event records containing data insertion, update, or deletion operations. The event record set is compared and analyzed with the preceding synchronization time of the target database. Based on the comparison and analysis results, unsynchronized transaction operation records are selected from the event record set to generate the target transaction operation set. The RPA robot simulates database query operations, generates a data extraction script based on the operation type and operation parameters in the target transaction operation set, and executes the data extraction script to pull the initial data set from the source database. Add transaction identifiers and timestamp metadata to the initial data set, and cache the initial data set to a distributed storage node.

3. The database multi-source heterogeneous data synchronization method according to claim 1, characterized in that, The step of detecting the task execution status of the distributed computing node, and activating the data synchronization channel and transmitting the target data set to the target database when the completion event of the target data set generation is detected, includes: The execution status parameters of each data shard task in the distributed computing node are obtained in real time through the RPA task status collector. The execution status parameters include at least the task progress percentage and the exception error code. When it is detected that the progress percentage of all data sharding tasks has reached the preset percentage and the error code is empty, a completion event for the generation of the target data set is generated, and the data synchronization channel running command is activated. According to the data synchronization channel operation instructions, a pre-configured channel connector is invoked to establish a bidirectional encrypted channel with the target database. The channel connector loads the corresponding communication protocol driver based on the type of the target database. The target data set in the distributed storage nodes is arranged into transmission sequences according to the fragment identifier order, and a flow control strategy for optimizing network congestion is injected. During transmission, a target verification mechanism is used to perform cyclic redundancy verification on each data packet fragment. If the verification fails, the retransmission process of the corresponding fragment is executed until all fragments are completely written into the buffer queue of the target database.

4. The database multi-source heterogeneous data synchronization method according to claim 3, characterized in that, The step of arranging the target data set in the distributed storage nodes according to the fragment identifier order for transmission sequence and injecting a flow control strategy for optimizing network congestion includes: Parse the fragment identifiers of each data fragment in the target dataset, extract the global sequence number field and fragment generation timestamp, generate the initial transmission sequence in ascending order of the global sequence number field value, and calculate the waiting transmission time of each data fragment based on the difference between the fragment generation timestamp and the current system timestamp. When the waiting transmission time exceeds the preset transmission time threshold, the priority weight of the corresponding data fragment is increased to the first preset level, and the real-time network parameter set between the distributed storage node and the target database is collected simultaneously. The network parameter set includes the normalized bandwidth utilization coefficient, latency attenuation coefficient and packet loss penalty coefficient. The priority weights are weighted and calculated with the normalized network parameters to generate a comprehensive transmission priority score for each data segment. The initial transmission sequence is reordered based on the score to generate an optimized transmission sequence. The transmission window size of each data segment in the transmission sequence is adjusted according to the ratio of the current available bandwidth to the data segment size. The data in the transmission sequence is fragmented and encapsulated into data packets carrying fragment identifiers and check codes through a preset flow control strategy. Parallel transmission channels are started, and the transmission delay and packet loss rate of each data packet are monitored in real time during the transmission process. When the packet loss rate of any data segment is detected to exceed the preset tolerance threshold, the transmission window size of the data segment is reduced to the second preset level according to the flow control strategy, and the transmission window size of the adjacent data segments is increased synchronously based on the current overall bandwidth utilization to maintain the balance of bandwidth utilization. After the target database returns a write confirmation signal carrying a fragment identifier, the transmission log status of the corresponding fragment identifier is updated; if the write confirmation signal of the target fragment identifier is not received within the preset timeout period, the corresponding data fragment is extracted from the distributed storage node based on the flow control strategy, repackaged, and inserted into the head of the transmission sequence for retransmission.

5. The database multi-source heterogeneous data synchronization method according to claim 1, characterized in that, The system performs conflict detection and identification based on the RPA conflict handling strategy configured in the target database. When a primary key conflict or unique constraint conflict is detected, the target database is updated based on preset conflict merging indication information, and a data synchronization log is generated, including: Configure a conflict detection rule set in the target database. The conflict detection rule set includes primary key duplicate detection rules, foreign key constraint verification rules, and unique index conflict rules. Create a data write transaction in the temporary storage area of ​​the target database, and sequentially apply the conflict detection rule set to perform conflict scanning on the target data set to generate a list of conflicting data. According to the conflict type in the conflict data list, the preset conflict merging instruction information is matched. The conflict merging instruction information includes timestamp overwrite strategy, field priority strategy and manual review strategy. The conflict type includes primary key conflict or unique constraint conflict. The RPA robot executes the automated operation corresponding to the conflict merging instruction information, updates the conflict data records in the target database, and writes the update results to the data synchronization log.

6. The database multi-source heterogeneous data synchronization method according to claim 5, characterized in that, The step of executing the automated operation corresponding to the conflict merging instruction information through the RPA robot, updating the conflict data records in the target database, and writing the update result to the data synchronization log includes: Based on the conflict type in the conflict data list, extract the source transaction timestamp and the current record timestamp of the target database from the conflict data record, perform time unit standardization conversion based on the source transaction timestamp and the current record timestamp, and generate time coverage priority parameters; By combining the pre-configured field priority mapping table, the normalized weight values ​​of the conflicting fields in the conflicting data records are extracted and field priority parameters are generated. The time coverage priority parameters and field priority parameters are weighted and merged according to a preset ratio to generate a comprehensive priority sorting sequence for each conflicting data record. Based on the comprehensive priority sorting sequence, traverse the conflicting data records: if there is a primary key conflict and the comprehensive priority exceeds the preset priority, generate a target record replacement script and perform an overwrite update; if there is a unique constraint conflict and the priority does not reach the preset priority, generate a conflicting field merging script based on field priority comparison, and retain the weight field data. After the script is executed, the RPA robot sends a verification request containing a transaction identifier to the target database. The returned actual data is compared with the expected value of the script field by field. If there is a difference, the difference record is re-inserted into the conflict data list and an alarm event is activated. The execution results of the target record replacement script or conflict field merging script, the field-by-field comparison results, and the transaction identifier are associated to generate a log entry with a version identifier; the log entry is synchronously written to the data synchronization log partition of the distributed storage node and the audit table of the target database, and an inverse index relationship with the original data shard is generated.

7. The database multi-source heterogeneous data synchronization method according to claim 1, characterized in that, The method further includes: During data synchronization, the running status of distributed computing nodes is monitored in real time. When a node failure or task timeout is detected, the unfinished tasks are automatically reassigned to healthy nodes. Alternatively, if a format parsing error occurs during the data conversion process, the RPA robot calls the backup conversion rule set to retry, and after the retry fails, the erroneous data fragment is extracted and sent to the repair queue; Alternatively, if a connection is interrupted during the writing process to the target database, the breakpoint position is automatically recorded and the writing operation is resumed from the breakpoint position after the connection is restored. Alternatively, periodically analyze error patterns in the data synchronization logs and adjust the data sharding task allocation strategy and conflict detection rule set to optimize synchronization efficiency.

8. A database multi-source heterogeneous data synchronization device, characterized in that, include: processor; A storage device having a computer program stored thereon, which, when executed by the processor, causes the processor to implement the database multi-source heterogeneous data synchronization method as described in any one of claims 1-7.

Images