Business data synchronization method, system and terminal based on distributed heterogeneous data sources

Abstract

The present application relates to a business data synchronization method, system and terminal based on distributed heterogeneous data sources, belonging to the technical field of data processing. The business data synchronization method includes: real-time monitoring of whether the business data in the business database of the current business node has changed; if so, determining whether the changed business data belongs to heterogeneous data; if so, converting the changed business data into standard business data; slicing the standard business data according to field sensitivity, which includes low sensitivity, medium sensitivity and high sensitivity; obtaining the slice length of each slice; encrypting each slice separately according to the slice length and sensitivity level; sending all encrypted slices to the target business node, which decrypts and restores the encrypted slices to update the stored historical business data according to the standard business data. The present application has the beneficial effect of reducing the latency of data synchronization.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to a method, system, and terminal for synchronizing business data based on distributed heterogeneous data sources. Background Art

[0002] With the rapid development of information technology, data has become a critical resource in modern society. Business data sources are becoming increasingly diverse within enterprise operations. Different departments and business systems may use different types of databases to store data, such as relational databases (such as MySQL, PostgreSQL, and Oracle) and non-relational databases (such as MongoDB and Redis), and even maintain a variety of data files. Furthermore, the widespread adoption of technologies such as the Internet, the Internet of Things, and Big Data has led to a continuous increase in the amount of data that enterprises must process, increasing data processing requirements. Enterprises need to integrate, cleanse, and transform this heterogeneous data for data analysis and mining, thereby supporting decision-making. However, the diversity of data sources, coupled with differences in data format, structure, and semantics, poses significant challenges to data synchronization. Data security and privacy protection also become key issues during data synchronization. Enterprises must ensure data security during transmission, storage, and processing to prevent risks such as data leakage and tampering. They must also comply with relevant laws and regulations to protect personal privacy.

[0003] Currently, common implementations of business data synchronization include scheduled tasks or batch processing. Scheduled tasks check and synchronize data in business databases at preset intervals. For example, every evening, a set time is set to copy all changed data from the source database to the target database. Batch processing synchronizes data all at once after a certain amount of changed data has accumulated.

[0004] Because scheduled tasks have fixed intervals, data synchronization may be delayed. In certain business scenarios, such as financial transactions and real-time monitoring, data real-time availability is crucial, and even the slightest delay can lead to serious consequences. Batch processing, which requires the accumulation of a certain amount of data, can also cause data synchronization delays. Summary of the Invention

[0005] In order to reduce the latency of data synchronization, the present application provides a business data synchronization method, system and terminal based on distributed heterogeneous data sources.

[0006] In a first aspect, the present application provides a method for synchronizing business data based on distributed heterogeneous data sources, which adopts the following technical solutions:

[0007] A business data synchronization method based on distributed heterogeneous data sources, comprising:

[0008] Real-time monitoring of whether the business data in the business database of the current business node has changed;

[0009] If so, determine whether the changed business data belongs to heterogeneous data;

[0010] If so, converting the changed business data into standard business data;

[0011] Slicing the standard business data according to field sensitivity, where the field sensitivity includes low sensitivity, medium sensitivity, and high sensitivity;

[0012] Get the slice length of each slice;

[0013] Encrypting each slice individually based on the slice length and sensitivity level;

[0014] All encrypted slices are sent to the target business node, and the target business node decrypts and restores the encrypted slices to update the stored historical business data according to the standard business data.

[0015] By implementing this technical solution, business data in the business database is monitored in real time for changes. Once a change is detected, subsequent processing is immediately performed, ensuring timely data updates across different business nodes and avoiding business decision errors or process disruptions caused by data delays. The system determines whether the changed business data is heterogeneous and converts the heterogeneous changed business data into standard business data. This allows data sources of different formats and structures to be synchronized under a unified standard, ensuring data consistency. Standard business data is sliced ​​and encrypted according to field sensitivity, enabling different encryption strategies to be adopted based on the sensitivity of the data. The encrypted slices are sent to the target business node, which decrypts and restores them. During transmission, the encrypted data effectively prevents data theft or tampering, ensuring data transmission security. After decryption and restoration at the target business node, historical business data can be updated based on the standard business data. The entire process is completed in a secure environment, ensuring the integrity and confidentiality of business data. Without waiting for fixed time intervals or accumulating a certain amount of data, data changes can be detected immediately, reducing data synchronization latency. Furthermore, by slicing standard business data according to high, medium, and low sensitivity levels, different encryption strategies can be adopted based on the data's sensitivity, precisely protecting data of varying sensitivities and effectively reducing the risk of data leakage. Each slice is encrypted separately based on its length and sensitivity level, ensuring the data in each slice remains independent after encryption. Even if the data in a slice is intercepted or tampered with during transmission, the security and integrity of the data in other slices will not be affected. Because the encryption key for each slice is independent, attackers cannot obtain data in other slices by cracking the encryption of one slice, further enhancing data security. Furthermore, after slicing standard business data, encryption operations can be performed in parallel. Compared to encrypting the entire standard business data at once, slicing reduces encryption complexity and improves efficiency. Especially for large-scale standard business data, slicing can fully utilize the computing power of multi-core processors to encrypt multiple slices simultaneously, significantly reducing encryption time.

[0016] Optionally, the step of converting the changed business data into standard business data includes:

[0017] Identifying the data type of the changed business data;

[0018] According to the data type, calling a preset data mapping table;

[0019] Retrieve standard business data templates;

[0020] According to the data type, the mapped changed business data is filled into the corresponding field of the standard business data template to form standard business data.

[0021] By employing the above technical solution, identifying the data type of changed business data is fundamental to the entire conversion process. Accurately identifying data types such as numbers, text, and dates ensures that subsequent data processing and conversion operations are based on the correct data attributes. Based on the identified data type, pre-set data mapping tables are retrieved to map changed business data from different sources and formats according to unified rules. Standard business data templates are retrieved and the mapped changed business data is populated into corresponding fields, ensuring that all business data follows the same structure and format. This helps eliminate issues caused by inconsistent data structures and facilitates data storage, querying, and analysis. All business data is organized according to standard business data templates, enabling unified methods and tools for data analysis, improving data processing efficiency and accuracy. Pre-set data mapping tables and standard business data templates can be reused across multiple data conversion processes. When new changed business data requires conversion, there's no need to re-define conversion rules; simply call the corresponding mapping table and template based on the data type to quickly complete the conversion, significantly reducing data processing time.

[0022] Optionally, the step of sending the encrypted slice to the target service node includes:

[0023] Determining a sending strategy according to the sensitivity level of the encrypted slice;

[0024] Sending the encrypted slice according to the sending strategy;

[0025] The sending strategy corresponding to the encrypted slice of high sensitivity level is to select accelerated channel sending, associate interference data for sending, and test the sending environment before sending;

[0026] The sending strategy corresponding to the encrypted slice of medium sensitivity level is to select a low-latency channel for sending and test the sending environment before sending;

[0027] The sending strategy corresponding to the encrypted slice of low sensitivity level is to select a batch buffer channel for sending.

[0028] By implementing the above technical solution, for highly sensitive encrypted slices, using accelerated channels for transmission reduces data transmission time across the network, lowering the risk of interception and attack. Transmitting data with associated interference increases data obfuscation, making it difficult for attackers to discern the true sensitive information from the large amount of interference data, even if the data is intercepted. Testing the sending environment before transmission can proactively identify potential security risks, such as network vulnerabilities and malware, ensuring data transmission in a secure environment. Using low-latency channels for sending encrypted slices with medium sensitivity ensures timely data synchronization and reduces data transmission risks. Testing the sending environment before transmission also ensures data security and prevents data tampering or leakage during transmission. Using batch buffer channels for sending encrypted slices with low sensitivity allows for centralized processing of multiple low-sensitivity data slices, reducing data transmission times and overhead, and improving overall data transmission efficiency. Selecting different transmission channels based on the sensitivity level of the encrypted slices ensures optimal allocation of network resources and reduces system processing burden.

[0029] Optionally, the step of sending the encrypted slice to the target service node further includes:

[0030] When sending, obtain the first estimated sending duration of the encrypted slice of the low sensitivity level, the second estimated sending duration of the encrypted slice of the medium sensitivity level, and the third estimated sending duration of the encrypted slice of the high sensitivity level;

[0031] The initial sending time of each encrypted slice is determined based on the first estimated sending time, the second estimated sending time, the third estimated sending time and the receiving time set by the target service node.

[0032] By adopting the above technical solution, by obtaining the estimated transmission duration of encrypted slices of different sensitivity levels and combining it with the reception time set by the target service node to determine the initial transmission time, each encrypted slice can be ensured to arrive at the target service node as close as possible to the target service node's expected time. When encrypted slices of different sensitivity levels arrive at the target service node at the same time, the target service node can process the data at once, avoiding waiting and processing delays caused by inconsistent data arrival times.

[0033] Optionally, the service data synchronization method further includes:

[0034] Determine whether the target service node has completely received each of the encrypted slices within the set receiving time;

[0035] If not, after decrypting the encrypted slice received by the target service node, determining whether the missing encrypted slice is an encrypted slice of a low sensitivity level;

[0036] If yes, restore the data of the encrypted slice with a low sensitivity level according to the received encrypted slice;

[0037] If not, extend the set receiving time and resend the missing encrypted slices to the target service node n times in a row.

[0038] The above technical solution enables timely detection of potential data transmission issues. In data synchronization scenarios involving distributed, heterogeneous data sources, factors such as network fluctuations and node failures can cause the loss of some encrypted slices. This check ensures that subsequent data processing is performed only after all necessary encrypted slices have been fully received, thus safeguarding the integrity of service data. If a missing encrypted slice is detected, the system further determines whether the missing encrypted slice is of low sensitivity. If so, the data is restored based on the received encrypted slice. Low-sensitivity data typically has less impact on critical business decisions. This restoration method can quickly restore data availability without disrupting normal business operations. If the missing encrypted slice is of high sensitivity, the set reception time is extended and the missing encrypted slice is resent to the target service node n times in a row. Through multiple retransmissions and extended reception times, critical data is ensured to be completely and accurately transmitted to the target service node, avoiding serious business losses caused by missing data. By promptly detecting and addressing missing encrypted slices, the risk of business interruption caused by incomplete data is reduced.

[0039] Optionally, the step of restoring data of the encrypted slice of a low sensitivity level according to the received encrypted slice includes:

[0040] Get the data length of the incomplete standard business data after restoration;

[0041] Searching for business data greater than the data length from the business database to construct a business data set;

[0042] Performing similarity matching between the incomplete business data and each historical business data in the business data set;

[0043] Filter out historical business data with similarity greater than the similarity threshold and sort them in descending order of priority;

[0044] Merging the filtered historical business data with the incomplete business data in sequence;

[0045] Obtain the semantic relevance after fusion;

[0046] The data of the encrypted slice with a low sensitivity level is restored based on the fused business data with the greatest semantic correlation.

[0047] By employing the above technical solution, by obtaining the length of the incomplete standard business data after restoration, searching the business database for business data greater than this length and constructing a business dataset, the data screening scope can be narrowed, focusing on business data that may match the incomplete data and avoiding interference from irrelevant data, thus laying the foundation for subsequent accurate restoration of the low-sensitivity encrypted slice. The incomplete business data is then matched against historical business data in the business dataset for similarity, and historical business data with similarity exceeding a threshold is selected. This allows precise location of historical data that is highly similar to the incomplete data. The data is then sorted from highest to lowest similarity and then fused sequentially, prioritizing the most closely matching data for fusion, improving data restoration accuracy. During the data fusion process, the semantic relevance of the fused data is determined, and the data of the low-sensitivity encrypted slice is restored based on the fused business data with the highest semantic relevance. This ensures that the restored data is semantically sound and coherent, avoiding semantic confusion or non-compliance with business logic, and ensuring that the restored data better meets actual business needs.

[0048] In a second aspect, the present application provides a business data synchronization system based on distributed heterogeneous data sources, which adopts the following technical solutions:

[0049] A business data synchronization system based on distributed heterogeneous data sources, comprising:

[0050] The monitoring module is used to monitor in real time whether the business data in the business database of the current business node has changed;

[0051] The judgment module is used to judge whether the changed business data belongs to heterogeneous data after the business data is changed;

[0052] a conversion module, configured to convert the changed business data into standard business data when the judgment module determines that the changed business data is yes;

[0053] A slicing encryption module, used to slice and encrypt the standard business data according to field sensitivity;

[0054] The sending module is used to send the encrypted slice to the target business node, and the target business node decrypts and restores the encrypted slice to update the stored historical business data according to the standard business data.

[0055] Optionally, the slice encryption module includes:

[0056] a slicing processing unit, configured to slice the standard business data according to field sensitivity, where the field sensitivity includes low sensitivity, medium sensitivity, and high sensitivity;

[0057] A length obtaining unit, used to obtain the slice length of each slice;

[0058] The slice encryption unit is used to encrypt each slice individually according to the slice length and sensitivity level.

[0059] In a third aspect, the present application provides a terminal that adopts the following technical solution:

[0060] A terminal, comprising:

[0061] A memory storing a business data synchronization program based on distributed heterogeneous data sources;

[0062] The processor is used to execute the program stored in the memory to implement the steps of the above-mentioned business data synchronization method based on distributed heterogeneous data sources.

[0063] In summary, this application has at least the following beneficial effects:

[0064] Business data in the business database is monitored in real time for changes. Once a change is detected, subsequent processing is immediately performed, ensuring timely data updates across different business nodes and avoiding business decision errors or process disruptions caused by data latency. The system determines whether the changed business data is heterogeneous and converts it into standard business data. This allows data sources of different formats and structures to be synchronized under a unified standard, ensuring data consistency. Standard business data is sliced ​​and encrypted based on field sensitivity, enabling different encryption strategies to be adopted based on data sensitivity. The encrypted slices are sent to the target business node, which decrypts and restores them. During transmission, the encrypted data effectively prevents data theft and tampering, ensuring data transmission security. After decryption and restoration at the target business node, historical business data can be updated based on the standard business data. The entire process is completed in a secure environment, ensuring the integrity and confidentiality of business data. Without waiting for fixed time intervals or accumulating a certain amount of data, data changes can be detected immediately, reducing data synchronization latency.

[0065] After slicing the standard business data, encryption operations can be performed in parallel. Compared with encrypting the entire standard business data at one time, slice encryption can reduce the complexity of encryption processing and improve encryption efficiency. Especially for large-scale standard business data, slice encryption can make full use of the computing power of multi-core processors and encrypt multiple slices at the same time, greatly shortening the time required for encryption. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a first flow chart of an embodiment of the present application;

[0067] Figure 2This is a second flow chart of an embodiment of the present application;

[0068] Figure 3 This is a third flow chart of an embodiment of the present application;

[0069] Figure 4 This is the fourth flow chart of the embodiment of the present application. DETAILED DESCRIPTION

[0070] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the following will be combined with the appended drawings of the embodiments of the present invention. Figure 1 -Attached Figure 4 The technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0071] The first embodiment of the present application discloses a method for synchronizing business data based on distributed heterogeneous data sources. Figure 1 The service data synchronization method may include S110-S170:

[0072] S110, real-time monitoring of whether the business data in the business database of the current business node has changed;

[0073] S120, if yes, determine whether the changed business data belongs to heterogeneous data;

[0074] S130, if yes, convert the changed business data into standard business data;

[0075] S140, slicing and encrypting standard business data according to field sensitivity;

[0076] S150: Send the encrypted slice to the target service node. The target service node decrypts and restores the encrypted slice to update the stored historical service data according to the standard service data.

[0077] If not, then the changed business data is sliced ​​and encrypted directly according to the field sensitivity.

[0078] S170, send the encrypted slice to the target business node, and the target business node decrypts and restores the encrypted slice to update the historical business data according to the changed business data.

[0079] Specifically, you can leverage database logging capabilities, such as MySQL's binary log (binlog), and use tools like Canal to parse the logs in real time to capture business data change events in the business database stored in the current business node. By defining a data structure template, the structure of the change data can be compared against the template. You can use a JSON schema to describe the data structure, then use a JSON validation library for comparison. For example, a standard product data structure template specifies that product information includes fields such as product_id, product_name, and price. If the captured change data contains an additional vendor_info field in addition to these fields, and this field does not exist in the standard template, the change data is considered heterogeneous data.

[0080] In addition, the specific methods of S160 and S170 are the same as those of S140 and S150.

[0081] Reference Figure 2 The steps of converting the changed business data into the standard business data include S210-S240:

[0082] S210, identifying the data type of the changed business data;

[0083] S220, calling a preset data mapping table according to the data type;

[0084] S230, retrieving a standard business data template;

[0085] S240 , filling the mapped changed business data into corresponding fields of the standard business data template according to the data type to form standard business data.

[0086] Specifically, data type detection functions in programming languages ​​can be used to identify data types. For structured data (such as data in a database table), the field data type can be obtained from database metadata. For unstructured data (such as JSON data), the data type can be determined based on the data's characteristics and context. For example, in Python, the type() function can be used to detect the data type. A preset data mapping table is stored in a database or configuration file. Using the identified data type as a key, the corresponding mapping table is retrieved from the storage. For example, a Python dictionary can be used to simulate this process, with the key being the data type and the value being the corresponding mapping table. Standard business data templates can be stored in a file (such as a JSON file) or database and read when needed. The Python json module can be used to read standard business data templates in JSON format. Then, the changed business data is traversed and the data is populated into the corresponding fields of the standard business data template based on the data type and mapping table.

[0087] For example, change the business data to:

[0088] Product name: "Smart Watch"

[0089] Price: 399

[0090] Is it in stock: Yes.

[0091] The standard business data template is as follows:

[0092] Product Name: [To be filled]

[0093] Sales price: [to be filled]

[0094] Stock Status: [To be filled].

[0095] According to the mapping relationship, "Product Name" corresponds to "Product Name" and "Smart Watch" is entered into the "Product Name" field. "Price" corresponds to "Selling Price" and "399" is entered into the "Selling Price" field. "Availability" corresponds to "Inventory Status" and "Yes" is entered into the "Inventory Status" field. The resulting standard business data is as follows: Product Name: Smart Watch, Sales Price: 399, Inventory Status: Yes.

[0096] Furthermore, the steps of slicing and encrypting standard business data according to field sensitivity include:

[0097] Standard business data is sliced ​​according to high sensitivity, medium sensitivity and low sensitivity; the slice length of each slice is obtained; and each slice is individually encrypted based on the slice length and sensitivity level.

[0098] Specifically, a field sensitivity rule table can be predefined and stored in a database or configuration file. The rule table contains field names and their corresponding sensitivity levels. When standard business data is received, the data fields are classified according to this rule table, and fields with the same sensitivity level are grouped together to form different slices. For example, for standard business data containing user and product information, "ID number" and "bank card number" are highly sensitive, "name" and "address" are medium sensitive, and "product name" and "price" are low sensitive. Based on these rules, the standard business data is divided into three slices: high sensitivity, medium sensitivity, and low sensitivity. For structured data (such as database records and JSON objects), the length of each slice can be obtained using the programming language's built-in functions. Taking Python's processing of JSON slices as an example, for a dictionary-formatted slice, the len() function can be used to obtain the number of key-value pairs. For example, if a highly sensitive slice contains two fields, "ID Number" and "Bank Card Number," the length of the highly sensitive slice is 2; if a medium-sensitive slice contains two fields, "Name" and "Address," the length of the medium-sensitive slice is 2; and if a low-sensitivity slice contains two fields, "Product Name" and "Price," the length of the low-sensitivity slice is 2. Different encryption algorithms can be used for different sensitivity levels. Highly sensitive data uses highly secure and computationally complex encryption algorithms, such as AES (Advanced Encryption Standard); medium-sensitive data can use the RSA algorithm; and low-sensitivity data can use simpler symmetric encryption algorithms, such as DES (Data Encryption Standard). Encryption parameters, such as key length, can be adjusted based on the slice length. In Python, encryption operations can be implemented using the cryptography library.

[0099] The steps of sending the encrypted slice to the target service node include:

[0100] Determine the sending strategy based on the sensitivity level of the encrypted slice; send the encrypted slice based on the sending strategy.

[0101] Specifically, the sending strategy corresponding to the encrypted slices with high sensitivity level is to select the accelerated channel for sending, and to send the associated interference data, and to test the sending environment before sending; the sending strategy corresponding to the encrypted slices with medium sensitivity level is to select the low-latency channel for sending, and to test the sending environment before sending; the sending strategy corresponding to the encrypted slices with low sensitivity level is to select the batch buffer channel for sending.

[0102] Specifically, for highly sensitive encrypted slices, methods such as data shuffling and data embedding can be used to mix interference data with encrypted slice data during transmission. Before sending a highly sensitive encrypted slice, a test packet is sent to the target service node to check network connectivity and data reception. Test metrics, such as packet loss rate and latency, can be set. Only when the test results meet pre-set standards will data transmission proceed. Network monitoring tools can be used to monitor and calculate latency for each channel in real time. The channels with the shortest latency are marked as low-latency channels. When medium-sensitivity encrypted slices are sent, the system automatically selects these low-latency channels for data transmission. Furthermore, the list of low-latency channels can be dynamically adjusted through regular network performance monitoring and evaluation. Similar to the pre-sentation testing for highly sensitive encrypted slices, a test packet is sent before sending medium-sensitivity encrypted slices to check network connectivity and the target node's reception capabilities. Test metrics can be adjusted appropriately based on the characteristics of medium-sensitivity data. For example, for highly sensitive encrypted slices, test metrics might include checking whether the network's packet loss rate is less than 1% and latency is within 100 milliseconds. For medium-sensitivity encrypted slices, test metrics might include checking whether the packet loss rate is less than 2% and latency is within 200 milliseconds. The purpose of the batch buffer channel is to centrally process multiple different types of data (including not only low-sensitivity encrypted slices but also other data) and then send them in batches to improve transmission efficiency. Low-sensitivity encrypted slices are first stored in a buffer. When the buffer reaches a certain amount of data or a certain time interval has passed, the data in the buffer is sent in batches to the target service node.

[0103] Before sending, obtain the first estimated sending time of the low-sensitivity encrypted slice, the second estimated sending time of the medium-sensitivity encrypted slice, and the third estimated sending time of the high-sensitivity encrypted slice; set the receiving time according to the first estimated sending time, the second estimated sending time, the third estimated sending time and the target service node to determine the initial sending time of each encrypted slice.

[0104] Specifically, for low-sensitivity encrypted slices, since they choose the batch buffer channel for transmission, the average transmission speed of the channel under different data volumes can be calculated based on historical data. According to the data volume of the current low-sensitivity encrypted slice and the average transmission speed, the first estimated transmission time is calculated. For example, past statistics show that the batch buffer channel can send an average of 10MB of data per second, and the current data volume of the low-sensitivity encrypted slice is 50MB, so the first estimated transmission time is 5 seconds. For medium-sensitivity encrypted slices, a low-latency channel is selected for transmission. The average transmission time of the channel can be obtained by simulating the transmission of medium-sensitivity encrypted slices of the same size multiple times in a test environment, recording the transmission time of each time, and taking the average value as the second estimated transmission time. For high-sensitivity encrypted slices, an accelerated channel is selected for transmission and associated with interference data. The theoretical bandwidth of the acceleration channel, the size of the interference data and the transmission rules can be comprehensively considered to estimate the third estimated transmission time by establishing a mathematical model. The mathematical model can be the third estimated transmission time ,in, The data size of the encrypted slice for high sensitivity level, is the size of the interference data, B is the theoretical bandwidth of the acceleration channel, and R is the correlation coefficient of the sending rule.

[0105] The target business node will set the receiving time in the system in advance, and the set time can be obtained through the system interface, or the system assigns the receiving time to the target node.

[0106] The initial sending time of each encrypted slice can be obtained by subtracting the estimated sending time of each encrypted slice from the receiving time set by the target business node.

[0107] For low-sensitivity encrypted slices, if the estimated duration of the first transmission is 5 seconds and the target service node sets the receiving time to "2025-06-20, 10:35:00", then the initial sending time of the low-sensitivity encrypted slice is "2025-06-20, 10:34:55".

[0108] For medium-sensitivity encrypted slices, if the second estimated sending time is 3 seconds and the target service node setting receiving time remains unchanged, then the initial sending time of the medium-sensitivity encrypted slice is "2025-06-20, 10:34:57".

[0109] For highly sensitive encrypted slices, if the estimated duration of the third transmission is 8 seconds and the target service node setting receiving time remains unchanged, then the initial transmission time of the highly sensitive encrypted slice is "2025-06-20, 10:34:52".

[0110] Furthermore, considering that factors such as network fluctuations may cause actual transmission times to differ from estimated times, an error buffer time can be set in other implementations. For example, a uniform error buffer time of 1 second is set, and data is sent 1 second in advance of the calculated initial transmission time to ensure that the data arrives before the reception time set by the target service node.

[0111] Reference Figure 3 The business data synchronization method further includes S310-S340:

[0112] S310, determining whether the target service node has completely received each encrypted slice within the set reception time;

[0113] S320: If not, after the target service node decrypts the received encrypted slice, it is determined whether the missing encrypted slice is a low-sensitivity encrypted slice;

[0114] S330: If yes, restore the data of the low-sensitivity encrypted slice according to the received encrypted slice;

[0115] S340: If not, extend the set receiving time and resend the missing encrypted slices to the target service node n times in a row.

[0116] Specifically, when sending encrypted slices, the identification information of all sent encrypted slices, such as numbers, hash values, etc., is recorded; each time the target business node receives an encrypted slice, it feeds back the identification information of the slice to the sender. The sender compares the received feedback information with the recorded identification information of all encrypted slices. If the fed-back identification information completely matches the recorded one within the set receiving time, it means that the target business node has completely received all the encrypted slices within the set receiving time; otherwise, it means that the reception is incomplete. When the reception is incomplete, the target business node checks the sensitivity level corresponding to the missing encrypted slice. If it is a low sensitivity level, the missing encrypted slice is determined to be a low sensitivity level encrypted slice; otherwise, it is not. When the missing encrypted slice is not a low sensitivity level encrypted slice, the set receiving time is extended for a certain period of time, for example, by 5 minutes, and then the missing encrypted slice is sent to the target business node n times in succession, for example, 3 times in succession.

[0117] Reference Figure 4 The step of restoring the data of the low-sensitivity encrypted slice according to the received encrypted slice includes S410-S470:

[0118] S410, obtaining the data length of the restored incomplete standard business data;

[0119] S420, searching for historical business data greater than the data length from the business database, and constructing a business data set;

[0120] S430, performing similarity matching between the incomplete business data and each historical business data in the business data set;

[0121] S440, filtering out historical business data with similarity greater than a similarity threshold, and sorting them in descending order of priority;

[0122] S450, sequentially integrating the filtered historical business data with the incomplete business data;

[0123] S460, obtaining the semantic relevance after fusion;

[0124] S470, restore the data of the low-sensitivity encrypted slice based on the fused business data with the greatest semantic relevance.

[0125] Specifically, when storing incomplete standard business data after restoration, its data length information can be recorded in a special metadata file or database table at the same time. When the data length needs to be obtained, the corresponding data length field value is read directly from the metadata file or database table. For example, if the relational database MySQL is used, a table named metadata can be created, in which data_id is used to identify incomplete standard business data, and the data_length field stores the corresponding data length. The required data length can be obtained by executing the SQL query statement SELECT data_length FROM metadata WHERE data_id ='specific data identifier'. Assuming that the business database is a PostgreSQL database, and the business data is stored in the business_data table, the table has a data_content field to store the business data content, and the data length can be obtained through the LENGTH(data_content) function. Use the SQL query statement 'The length of the data obtained in step S410' is used to search for business data that meets the conditions. The query results are stored in a temporary table or memory data structure (such as a list in Python) to construct a business data set. The edit distance (Levenshtein distance) algorithm is used to calculate the similarity. In Python, the python-Levenshtein library can be used. Pass the incomplete business data and the business data set as parameters to the function to obtain the similarity matching result. For example, if the similarity threshold is set to 0.6, the results of step S430 are combined in Python to filter and sort, filter out historical business data with a similarity greater than the threshold, and sort them from large to small by similarity. The filtered historical business data and the incomplete business data are sequentially spliced ​​together to obtain fused data. Then, a pre-trained model in natural language processing (NLP), such as the BERT model, is used, which can be implemented in Python with the help of the transformers library. The fused data is input into the trained BERT model for encoding, and then the cosine similarity between the encoded vectors is calculated to represent the semantic relevance, and then the fused business data corresponding to the maximum semantic relevance is found; after the fused business data is sliced ​​according to field sensitivity, the low-sensitivity slice data is the data corresponding to the missing low-sensitivity encrypted slice.

[0126] The implementation principle of this embodiment is:

[0127] Monitor in real time whether the business data in the business database has changed. If so, further determine whether the changed business data belongs to heterogeneous data. If so, convert the changed business data into standard business data, and then slice and encrypt the standard business data according to field sensitivity; then obtain the first estimated sending time of the low-sensitivity encrypted slice, the second estimated sending time of the medium-sensitivity encrypted slice, and the third estimated sending time of the high-sensitivity encrypted slice, and set the receiving time according to the first estimated sending time, the second estimated sending time, the third estimated sending time and the target business node to determine the initial sending time of each encrypted slice; after sending according to the initial sending time, determine whether the target business node has completely received each encrypted slice within the set receiving time. If not, after the target business node decrypts the received encrypted slice, determine whether the missing encrypted slice is a low-sensitivity encrypted slice. If so, restore the data of the low-sensitivity encrypted slice based on the received encrypted slice, and the target business node updates the historical business data based on the complete standard business data, thereby achieving data synchronization.

[0128] Based on the above method embodiments, the second embodiment of the present application discloses a business data synchronization system based on distributed heterogeneous data sources. The business data synchronization system based on distributed heterogeneous data sources in the embodiment of the present application can implement any of the above-mentioned business data synchronization methods based on distributed heterogeneous data sources, and the specific working processes of each module in the business data synchronization system based on distributed heterogeneous data sources can refer to the corresponding processes in the above-mentioned method embodiments.

[0129] For ease of understanding, an example is given below: A business data synchronization system based on distributed heterogeneous data sources includes:

[0130] The monitoring module is used to monitor in real time whether the business data in the business database of the current business node has changed;

[0131] The judgment module is used to judge whether the changed business data belongs to heterogeneous data after the business data is changed;

[0132] The conversion module is used to convert the changed business data into standard business data when the judgment module determines that the changed business data is yes;

[0133] Slice encryption module, used to slice and encrypt standard business data according to field sensitivity;

[0134] The sending module is used to send the encrypted slice to the target business node, and the target business node decrypts and restores the encrypted slice to update the stored historical business data according to the standard business data.

[0135] The slice encryption module may include:

[0136] Slicing processing unit, used to slice standard business data according to field sensitivity, which includes low sensitivity, medium sensitivity and high sensitivity;

[0137] A length obtaining unit, used to obtain the slice length of each slice;

[0138] The slice encryption unit is used to encrypt each slice individually according to the slice length and sensitivity level.

[0139] The third embodiment of the present application provides a terminal. As an implementation of the terminal, the terminal may include: a memory and a processor; wherein,

[0140] The memory is used to store business data synchronization programs based on distributed heterogeneous data sources;

[0141] The processor is used to execute the program stored in the memory to implement the steps of the above-mentioned business data synchronization method based on distributed heterogeneous data sources.

[0142] The memory may be communicatively connected to the processor via a communication bus, and the communication bus may be an address bus, a data bus, a control bus, or the like.

[0143] In addition, the memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0144] The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0145] The above are all preferred embodiments of the present application and are not intended to limit the scope of protection of the present application. Unless otherwise specified, any feature disclosed in this specification (including the abstract and drawings) may be replaced by other equivalent or similar features. In other words, unless otherwise specified, each feature is merely an example of a series of equivalent or similar features.

Claims

1. A business data synchronization method based on distributed heterogeneous data sources, characterized in that: include: Real-time monitoring of whether the business data in the business database of the current business node has changed; If so, determine whether the changed business data belongs to heterogeneous data; If so, converting the changed business data into standard business data; Slicing the standard business data according to field sensitivity, where the field sensitivity includes low sensitivity, medium sensitivity, and high sensitivity; Get the slice length of each slice; Encrypting each slice individually based on the slice length and sensitivity level; Send all encrypted slices to the target service node, and the target service node decrypts and restores the encrypted slices to update the stored historical service data according to the standard service data; The steps of sending all encrypted slices to the target service node include: Determining a sending strategy according to the sensitivity level of the encrypted slice; Sending the encrypted slice according to the sending strategy; The sending strategy corresponding to the encrypted slice of high sensitivity level is to select accelerated channel sending, associate interference data for sending, and test the sending environment before sending; The sending strategy corresponding to the encrypted slice of medium sensitivity level is to select a low-latency channel for sending and test the sending environment before sending; The sending strategy corresponding to the encrypted slice of low sensitivity level is to select a batch buffer channel for sending; When sending, obtain the first estimated sending duration of the encrypted slice of the low sensitivity level, the second estimated sending duration of the encrypted slice of the medium sensitivity level, and the third estimated sending duration of the encrypted slice of the high sensitivity level; The first estimated sending duration is: Based on historical data statistics of the average sending speed of the batch buffer channel under different data volumes, the estimated duration of the first transmission is calculated according to the data volume of the current low-sensitivity encrypted slice and the average sending speed; The second estimated sending duration is: By simulating the sending of medium-sensitivity encrypted slices of the same size multiple times in a test environment, recording the sending time each time, and taking the average value to obtain the average sending time of the low-latency channel, this is used as the estimated second sending time; The third estimated sending duration is: Estimate the estimated duration of the third transmission based on the theoretical bandwidth of the acceleration channel, the size of the interference data, and the transmission rule coefficient; The initial sending time of each encrypted slice is determined based on the first estimated sending time, the second estimated sending time, the third estimated sending time and the receiving time set by the target service node.

2. A method for synchronizing business data based on distributed heterogeneous data sources according to claim 1, characterized in that: The step of converting the changed business data into standard business data includes: Identifying the data type of the changed business data; According to the data type, calling a preset data mapping table; Retrieve standard business data templates; According to the data type, the mapped changed business data is filled into the corresponding field of the standard business data template to form standard business data.

3. A method for synchronizing business data based on distributed heterogeneous data sources according to claim 1, characterized in that: The business data synchronization method further includes: Determine whether the target service node has completely received each of the encrypted slices within the set receiving time; If not, after decrypting the encrypted slice received by the target service node, determining whether the missing encrypted slice is an encrypted slice of a low sensitivity level; If yes, restore the data of the encrypted slice with a low sensitivity level according to the received encrypted slice; If not, extend the set receiving time and resend the missing encrypted slices to the target service node n times in a row.

4. A method for synchronizing business data based on distributed heterogeneous data sources according to claim 3, characterized in that: The step of restoring data of the encrypted slice of low sensitivity level according to the received encrypted slice includes: Get the data length of the incomplete standard business data after restoration; Searching for business data greater than the data length from the business database to construct a business data set; Performing similarity matching between the incomplete business data and each historical business data in the business data set; Filter out historical business data with similarity greater than the similarity threshold and sort them in descending order of priority; Merging the filtered historical business data with the incomplete business data in sequence; Obtain the semantic relevance after fusion; The data of the encrypted slice with a low sensitivity level is restored based on the fused business data with the greatest semantic correlation.

5. A business data synchronization system based on distributed heterogeneous data sources, characterized in that: Executing the business data synchronization method based on distributed heterogeneous data sources according to any one of claims 1 to 4, comprising: The monitoring module is used to monitor in real time whether the business data in the business database of the current business node has changed; The judgment module is used to judge whether the changed business data belongs to heterogeneous data after the business data is changed; a conversion module, configured to convert the changed business data into standard business data when the judgment module determines that the changed business data is yes; A slicing encryption module, used to slice and encrypt the standard business data according to field sensitivity; The sending module is used to send the encrypted slice to the target business node, and the target business node decrypts and restores the encrypted slice to update the stored historical business data according to the standard business data.

6. A business data synchronization system based on distributed heterogeneous data sources according to claim 5, characterized in that: The slice encryption module includes: a slicing processing unit, configured to slice the standard business data according to field sensitivity, where the field sensitivity includes low sensitivity, medium sensitivity, and high sensitivity; A length obtaining unit, used to obtain the slice length of each slice; The slice encryption unit is used to encrypt each slice individually according to the slice length and sensitivity level.

7. A terminal, characterized in that: include: A memory storing a business data synchronization program based on distributed heterogeneous data sources; A processor is used to execute the program stored in the memory to implement the steps of the business data synchronization method based on distributed heterogeneous data sources as described in any one of claims 1 to 4.

Images