Remote synchronization method and system for database data

Abstract

The invention discloses a remote synchronization method and system for database data, and relates to the technical field of data processing, and the method comprises the steps: collecting the processing information data of a data block, obtaining the processing data of the data block, and carrying out the analysis and calculation in combination with an importance index and a data security index to obtain a synchronous transmission index of the data block; according to the method, the data security index model is constructed by monitoring the bandwidth, delay and packet loss rate of the receiving end in real time, the network transmission reliability is dynamically evaluated, and transmission strategy adjustment is automatically triggered when network fluctuation or high packet loss rate is detected, so that data loss caused by forced synchronization in a weak network environment is prevented, and the network transmission reliability is improved. In addition, a synchronous transmission index is calculated in combination with an importance index and a security index, data is divided into a high priority and a low priority, the high priority is used for real-time transmission through a private line network to ensure core service continuity, and the low priority is used for transmission in a low peak period by adopting a batch compression technology to reduce network load.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a remote synchronization method and system for database data. Background Art

[0002] As the core infrastructure supporting modern distributed systems and cloud computing architecture, the development history and technological evolution of database remote synchronization technology are closely related to the deep-seated needs of enterprise digital transformation. Early single-database architectures gradually exposed bottlenecks under the surge in data volume and access pressure, prompting master-slave replication and read-write separation technologies to become mainstream solutions. By concentrating write operations on the master database and distributing read operations to slave databases, single-point pressure is effectively alleviated. However, with the increase in business complexity, vertical and horizontal sharding architectures have further promoted the development of distributed databases. Middleware such as MyCat shields the differences in underlying data sources and realizes transparent processing of cross-database queries and distributed transactions, laying the architectural foundation for database synchronization technology.

[0003] However, in the remote synchronization process of traditional databases, the data of the source database is directly transferred to the receiving end for synchronization. However, this method does not take into account the importance of the data and the network conditions of the receiving end, which may result in important data not being synchronized in a timely manner, or data synchronization when the network security is poor, resulting in risks such as data leakage.

[0004] In view of the above technical defects, a solution is now proposed. Summary of the Invention

[0005] The purpose of the present invention is to solve the problem that in the remote synchronization process of traditional databases, the data of the source database is directly transmitted to the receiving end for synchronization. However, this method does not take into account the importance of the data and the network conditions of the receiving end, which may result in important data not being synchronized in time, or data synchronization when the network security is poor, resulting in risks such as data leakage.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: a remote synchronization method of database data, comprising the following steps:

[0007] Step 1: Compare the update database with the source database to find the difference between the two, identify the data that needs to be updated from the source database, and integrate this data into a synchronized data set. The data in the synchronized data set is divided into blocks to form multiple groups of data blocks;

[0008] Step 2: Collect information data of the data block to obtain importance data of the data block, and analyze and calculate the importance index of the data block. The importance index is used to reflect the importance of the data block;

[0009] Step 3: Collect information data from the data synchronization receiving end to obtain secure transmission data of the data synchronization receiving end, and perform analysis and calculation to obtain a data security index of the data synchronization receiving end. The data security index is used to reflect the security of the data block transmission process;

[0010] Step 4: Collect the processing information data of the data block to obtain the data block processing data, and analyze and calculate the synchronous transmission index of the data block in combination with the importance index and the data security index. The synchronous transmission index of the data block is used to reflect the priority of the data block transmission process.

[0011] Furthermore, the importance data includes the update frequency, access frequency and size of the data block, the secure transmission data includes the bandwidth value, network delay and packet loss rate of the data synchronization receiving end, and the data block processing data includes the compression rate and encryption strength of the data block.

[0012] Furthermore, the calculation process of the importance index of the data block is as follows:

[0013] S11, obtaining the update frequency, access frequency and size data of the data block and performing analysis and calculation;

[0014] S12. Calculate the importance index imp of the data block according to the following formula:

[0015]

[0016] Among them, F is the update frequency of the data block, F e is the standard update frequency of the preset data block, f is the access frequency of the data block, and f max is the maximum access frequency of the data block, f min is the minimum access frequency of the data block, α is the preset update weight coefficient of the data block, β is the access weight coefficient of the data block, S is the size of the data block, and the larger the value of the importance index of the data block, the more important the data block is.

[0017] Furthermore, the calculation process of the data security index of the data synchronization receiving end is as follows:

[0018] S21. Obtain the bandwidth value, network delay, and packet loss rate data of the data synchronization receiving end and perform analysis and calculation;

[0019] S22. Calculate the data security index safe at the data synchronization receiving end according to the following formula:

[0020]

[0021] Among them, B is the available bandwidth value of the data synchronization receiving end, B maxis the maximum bandwidth value of the data synchronization receiving end, a is the preset bandwidth weight coefficient, T max is the maximum network delay of the data synchronization receiving end, T is the average network delay of the data synchronization receiving end, b is the preset weight coefficient of the network delay, P avg is the packet loss rate of the data synchronization receiving end, c is the preset weight coefficient of data packet loss, and the larger the data security index value of the data synchronization receiving end, the safer the data block transmission;

[0022] S23. Obtain the preset data security threshold safe th , compared with the data security index safe, when safe>safe th , it means that the data transmission security of the data synchronization receiving end is high and data blocks can be transmitted.

[0023] Furthermore, the calculation process of the synchronous transmission index of the data block is as follows:

[0024] S31. Obtain compression ratio and encryption strength data of the data block and perform analysis and calculation in combination with importance index and data security index;

[0025] S32. Calculate the synchronous transmission index Sync of the data block according to the following formula:

[0026]

[0027] Among them, imp is the importance index of the data block, safe is the data security index of the data synchronization receiving end, CR is the compression ratio of the data block, and CR max is the maximum compression ratio of the data block, CR min is the minimum compression ratio of the data block, EFS is the encryption strength of the data block, e The standard encryption strength for the preset data block;

[0028] S33 obtains the synchronous transmission index Sync of each data block, sorts them in descending order, generates a data block remote synchronization work order, and obtains the preset synchronous transmission threshold Sync e , compared with the synchronous transmission index Sync, Sync>Sync e , divided into high priority data blocks, Sync≤Sync e , divided into low-priority data blocks, and different transmission strategies are adopted according to data blocks of different priorities.

[0029] The present invention also provides a remote synchronization system for database data, comprising: a data comparison unit, a data acquisition unit, an important analysis unit, a security analysis unit and a synchronous transmission unit;

[0030] The data comparison unit is used to obtain data information from the update database and the source database for comparison, find out the difference data between the two, identify the data that needs to be updated from the source database, and then divide the data that needs to be updated into blocks to form data blocks;

[0031] The data acquisition unit is used to obtain importance data, security transmission data and data block processing data through the network database, and send the importance data to the importance analysis unit, send the security transmission data to the security analysis unit, and send the data block processing data to the synchronous transmission unit respectively;

[0032] The importance analysis unit is used to receive importance data, perform analysis and calculation, and then obtain the importance index of the data block, which is used to reflect the importance of the data block;

[0033] The security analysis unit is used to receive the secure transmission data, analyze and calculate the data security index of the data synchronization receiving end, and compare it with the preset data security threshold to obtain the data transmission security of the data synchronization receiving end;

[0034] The synchronous transmission unit is used to receive data blocks and process data, and analyze and calculate the synchronous transmission index of the data block in combination with the importance index and data security index. By sorting the numerical values ​​of the synchronous transmission index of each data block, different data blocks are intelligently transmitted.

[0035] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0036] This remote synchronization method and system for database data collects update frequency, access frequency and data block size through a data importance classification mechanism, and calculates the importance index through a quantitative formula, so that the system can accurately identify high-value data and prioritize the allocation of transmission resources, effectively preventing important business data from being affected by synchronization lags and affecting decision-making response speed. At the same time, a data security index model is constructed by real-time monitoring of the receiving end bandwidth, delay and packet loss rate, and network transmission reliability is dynamically evaluated. When network fluctuations or high packet loss rates are detected, transmission strategy adjustments are automatically triggered, preventing data loss caused by forced synchronization in a weak network environment and avoiding the transmission of sensitive information during non-safe periods. In addition, the importance index and security index are combined to calculate the synchronization transmission index, and data is divided into high priority and low priority. The former is transmitted in real time through a dedicated network to ensure the continuity of core business, and the latter uses batch compression technology to transmit during off-peak hours to reduce network load. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Shown is a schematic flow chart of the method of the present invention;

[0038] Figure 2 Shown is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0040] Example 1:

[0041] like Figure 1 As shown, a remote synchronization method for database data first compares the updated database with the source database, determines the incremental data (newly added, modified, and deleted records) that need to be synchronized through comparison tools (such as SQL Server's CHECKSUM and MySQL's pt-table-checksum), finds the difference data between the two, identifies the data that needs to be updated from the source database, and integrates this data into a synchronization data set. The data in the synchronization data set is divided into blocks according to business logic to form multiple groups of data blocks.

[0042] Then, the information data of the data block is collected to obtain the importance data of the data block. The importance data includes the update frequency, access frequency and size of the data block. The importance index of the data block is obtained by analysis and calculation. The importance index is used to reflect the importance of the data block.

[0043] The calculation process of the importance index of a data block is as follows:

[0044] S11, obtaining the update frequency, access frequency and size data of the data block and performing analysis and calculation;

[0045] S12. Calculate the importance index imp of the data block according to the following formula:

[0046]

[0047] Among them, F is the update frequency of the data block, F e is the standard update frequency of the preset data block, f is the access frequency of the data block, and f max is the maximum access frequency of the data block, f min is the minimum access frequency of the data block, α is the preset update weight coefficient of the data block, β is the access weight coefficient of the data block, S is the size of the data block, and the larger the value of the importance index of the data block, the more important the data block is.

[0048] Afterwards, the information data of the data synchronization receiving end is collected to obtain the data synchronization receiving end's secure transmission data. The secure transmission data includes the bandwidth value, network delay and packet loss rate of the data synchronization receiving end. The data security index of the data synchronization receiving end is analyzed and calculated to reflect the security of the data block transmission process.

[0049] The calculation process of the data security index at the data synchronization receiving end is as follows:

[0050] S21. Obtain the bandwidth value, network delay, and packet loss rate data of the data synchronization receiving end and perform analysis and calculation;

[0051] S22. Calculate the data security index safe at the data synchronization receiving end according to the following formula:

[0052]

[0053] Among them, B is the available bandwidth value of the data synchronization receiving end, B max is the maximum bandwidth value of the data synchronization receiving end, a is the preset bandwidth weight coefficient, T max is the maximum network delay of the data synchronization receiving end, T is the average network delay of the data synchronization receiving end, b is the preset weight coefficient of the network delay, P avg is the packet loss rate of the data synchronization receiving end, c is the preset weight coefficient of data packet loss, and a+b+c=1. The larger the data security index value of the data synchronization receiving end, the safer the data block transmission;

[0054] S23. Obtain the preset data security threshold safe th , compared with the data security index safe, when safe>safe th , it means that the data transmission security of the data synchronization receiving end is high and the data block can be transmitted. When safe≤safe th , it means that the data transmission security of the data synchronization receiving end is low, and the data blocks are prone to being changed and leaked during the transmission process. The transmission of data blocks should be prohibited, and staff should be dispatched to conduct a comprehensive inspection of the security of network transmission.

[0055] Finally, the data block processing information data is collected to obtain the data block processing data, which includes the compression rate and encryption strength of the data block. The data block processing data is combined with the importance index and data security index to analyze and calculate the data block synchronization transmission index. The data block synchronization transmission index is used to reflect the priority of the data block transmission process.

[0056] The calculation process of the synchronous transmission index of a data block is as follows:

[0057] S31. Obtain compression ratio and encryption strength data of the data block and perform analysis and calculation in combination with importance index and data security index;

[0058] S32. Calculate the synchronous transmission index Sync of the data block according to the following formula:

[0059]

[0060] Among them, imp is the importance index of the data block, safe is the data security index of the data synchronization receiving end, CR is the compression ratio of the data block, and CR max is the maximum compression ratio of the data block, CR min is the minimum compression ratio of the data block, EFS is the encryption strength of the data block, e The standard encryption strength for the preset data block;

[0061] S33 obtains the synchronous transmission index Sync of each data block, sorts them in descending order, generates a data block remote synchronization work order, and obtains the preset synchronous transmission threshold Sync e , compared with the synchronous transmission index Sync, Sync>Sync e , divided into high priority data blocks, Sync≤Sync e , divided into low-priority data blocks, low-latency channels (such as dedicated network) are used for high-priority data blocks, and batch compression transmission is used for low-priority data blocks (such as low-peak hours at night). It should be noted that if a data block fails to be transmitted, the system will automatically retry 3 times; if it still fails, it will be marked as an exception and the operation and maintenance personnel will be notified.

[0062] Example 2:

[0063] like Figure 2 As shown, a remote synchronization system for database data includes: a data comparison unit, a data acquisition unit, an important analysis unit, a security analysis unit and a synchronous transmission unit;

[0064] The data comparison unit is used to obtain data information from the update database and the source database for comparison, find out the difference data between the two, identify the data that needs to be updated from the source database, and then divide the data that needs to be updated into blocks to form data blocks;

[0065] The data acquisition unit is used to obtain importance data, security transmission data and data block processing data through the network database, and send the importance data to the importance analysis unit, send the security transmission data to the security analysis unit, and send the data block processing data to the synchronous transmission unit respectively;

[0066] The importance analysis unit is used to receive importance data, perform analysis and calculation, and then obtain the importance index of the data block, which is used to reflect the importance of the data block;

[0067] The security analysis unit is used to receive the secure transmission data, analyze and calculate the data security index of the data synchronization receiving end, and compare it with the preset data security threshold to obtain the data transmission security of the data synchronization receiving end;

[0068] The synchronous transmission unit is used to receive data blocks and process data, and analyze and calculate the synchronous transmission index of the data block in combination with the importance index and data security index. By sorting the numerical values ​​of the synchronous transmission index of each data block, different data blocks are intelligently transmitted.

[0069] The present invention collects update frequency, access frequency and data block size through a data importance classification mechanism, and calculates the importance index through a quantitative formula, so that the system can accurately identify high-value data and prioritize the allocation of transmission resources, effectively avoiding the impact of synchronization lag on important business data on decision-making response speed. At the same time, a data security index model is constructed by real-time monitoring of the receiving end bandwidth, delay and packet loss rate, and the network transmission reliability is dynamically evaluated. When network fluctuations or high packet loss rates are detected, the transmission strategy adjustment is automatically triggered, which not only prevents data loss caused by forced synchronization in a weak network environment, but also avoids the transmission of sensitive information during non-safe periods. In addition, the synchronization transmission index is calculated based on the importance index and the security index, and the data is divided into high priority and low priority. The former is transmitted in real time through a dedicated network to ensure the continuity of core business, and the latter uses batch compression technology to transmit during off-peak hours to reduce network load.

[0070] The size of the interval and threshold is set to facilitate comparison. The size of the threshold depends on the amount of sample data and the number of bases set by technical personnel in this field for each set of sample data; as long as it does not affect the proportional relationship between the parameter and the quantized value.

[0071] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formulas are set by those skilled in the art according to actual conditions.

[0072] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A remote synchronization method for database data, characterized in that: The following steps are involved: Step 1: Compare the update database with the source database to find the difference between the two, identify the data that needs to be updated from the source database, and integrate this data into a synchronized data set. The data in the synchronized data set is divided into blocks to form multiple groups of data blocks; Step 2: Collect information data of the data block to obtain importance data of the data block, and analyze and calculate the importance index of the data block. The importance index is used to reflect the importance of the data block; Step 3: Collect information data from the data synchronization receiving end to obtain secure transmission data of the data synchronization receiving end, and perform analysis and calculation to obtain a data security index of the data synchronization receiving end. The data security index is used to reflect the security of the data block transmission process; Step 4: Collect the processing information data of the data block to obtain the data block processing data, and analyze and calculate the synchronous transmission index of the data block in combination with the importance index and the data security index. The synchronous transmission index of the data block is used to reflect the priority of the data block transmission process.

2. A remote synchronization method for database data according to claim 1, characterized in that: The importance data includes the update frequency, access frequency and size of the data block; the secure transmission data includes the bandwidth value, network delay and packet loss rate of the data synchronization receiving end; and the data block processing data includes the compression rate and encryption strength of the data block.

3. A remote synchronization method for database data according to claim 1, characterized in that: The calculation process of the importance index of a data block is as follows: S11, obtaining the update frequency, access frequency and size data of the data block and performing analysis and calculation; S12. Calculate the importance index imp of the data block according to the following formula: Among them, F is the update frequency of the data block, F e is the standard update frequency of the preset data block, f is the access frequency of the data block, and f max is the maximum access frequency of the data block, f min is the minimum access frequency of the data block, α is the preset update weight coefficient of the data block, β is the access weight coefficient of the data block, S is the size of the data block, and the larger the value of the importance index of the data block, the more important the data block is.

4. A remote synchronization method for database data according to claim 1, characterized in that: The calculation process of the data security index at the data synchronization receiving end is as follows: S21. Obtain the bandwidth value, network delay, and packet loss rate data of the data synchronization receiving end and perform analysis and calculation; S22. Calculate the data security index safe at the data synchronization receiving end according to the following formula: Among them, B is the available bandwidth value of the data synchronization receiving end, B max is the maximum bandwidth value of the data synchronization receiving end, a is the preset bandwidth weight coefficient, T max is the maximum network delay of the data synchronization receiving end, T is the average network delay of the data synchronization receiving end, b is the preset weight coefficient of the network delay, P avg is the packet loss rate of the data synchronization receiving end, c is the preset weight coefficient of data packet loss, and the larger the data security index value of the data synchronization receiving end, the safer the data block transmission; S23. Obtain the preset data security threshold safe th , compared with the data security index safe, when safe>safe th , it means that the data transmission security of the data synchronization receiving end is high and data blocks can be transmitted.

5. The remote synchronization method of database data according to claim 1, characterized in that: The calculation process of the synchronous transmission index of a data block is as follows: S31. Obtain compression ratio and encryption strength data of the data block and perform analysis and calculation in combination with importance index and data security index; S32. Calculate the synchronous transmission index Sync of the data block according to the following formula: Among them, imp is the importance index of the data block, safe is the data security index of the data synchronization receiving end, CR is the compression ratio of the data block, and CR max is the maximum compression ratio of the data block, CR min is the minimum compression ratio of the data block, EFS is the encryption strength of the data block, e The standard encryption strength for the preset data block; S33 obtains the synchronous transmission index Sync of each data block, sorts them in descending order, generates a data block remote synchronization work order, and obtains the preset synchronous transmission threshold Sync e , compared with the synchronous transmission index Sync, Sync>Sync e , divided into high priority data blocks, Sync≤Sync e , divided into low-priority data blocks, and different transmission strategies are adopted according to data blocks of different priorities.

6. A remote synchronization system for database data, characterized in that: include: Data comparison unit, data collection unit, important analysis unit, safety analysis unit and synchronous transmission unit; The data comparison unit is used to obtain data information from the update database and the source database for comparison, find out the difference data between the two, identify the data that needs to be updated from the source database, and then divide the data that needs to be updated into blocks to form data blocks; The data acquisition unit is used to obtain importance data, security transmission data and data block processing data through the network database, and send the importance data to the importance analysis unit, send the security transmission data to the security analysis unit, and send the data block processing data to the synchronous transmission unit respectively; The importance analysis unit is used to receive importance data, perform analysis and calculation, and then obtain the importance index of the data block, which is used to reflect the importance of the data block; The security analysis unit is used to receive the secure transmission data, analyze and calculate the data security index of the data synchronization receiving end, and compare it with the preset data security threshold to obtain the data transmission security of the data synchronization receiving end; The synchronous transmission unit is used to receive data blocks and process data, and analyze and calculate the synchronous transmission index of the data block in combination with the importance index and data security index. By sorting the numerical values ​​of the synchronous transmission index of each data block, different data blocks are intelligently transmitted.

Images