A method for efficiently pushing data in a mass data synchronization process

Abstract

The application discloses a kind of in mass data synchronization process efficiently push data's method, belong to data push method field, comprising the following steps: step one: data sender carries out segmentation slice to the data to be pushed, forms segmented data block;Step two: according to the preset encryption rule, segmented data block is disordered;Step three: segmented data block is pushed to data receiver by multithread concurrent running;Step four: data receiver is combined segmented data block according to the preset decryption rule.This application, the data to be pushed is pushed to data receiver by segmentation slice means, multithread concurrent running, finally data receiver combines segmented data block, reaches the purpose of efficient data synchronization.

Description

Technical Field

[0001] This invention relates to the field of data push methods, specifically a method for efficiently pushing data during the synchronization of massive amounts of data. Background Technology

[0002] Data synchronization refers to the real-time or periodic updating and matching of data from two or more different locations, systems, or devices to maintain data consistency and integrity. Data synchronization improves data reliability, security, availability, and sharing, providing strong support for business decisions and data analysis.

[0003] Data synchronization requires data push. While this is manageable for small datasets, synchronizing massive amounts of data presents significant challenges, the most common being data transmission speed. Due to the sheer volume of data, transmission speed often becomes a critical factor limiting synchronization efficiency. Network bandwidth, storage device read / write speeds, and the quality of the synchronization algorithm all directly impact transmission speed. Therefore, improving the data push speed during massive data synchronization has become an urgent problem to solve.

[0004] Therefore, those skilled in the art have provided a method for efficiently pushing data during the synchronization of massive amounts of data, in order to solve the problems mentioned in the background art above. Summary of the Invention

[0005] The purpose of this invention is to provide a method for efficiently pushing data during the synchronization of massive amounts of data. The method involves segmenting and slicing the data to be pushed, and then using multiple threads to push the data to the data receiver concurrently. Finally, the data receiver combines the segmented data blocks to achieve efficient data synchronization, thereby solving the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A method for efficiently pushing data during massive data synchronization includes the following steps:

[0008] Step 1: The data sender segments the data to be pushed into data blocks.

[0009] Step 2: Shuffle the segmented data blocks according to the preset encryption rules;

[0010] Step 3: Push segmented data blocks to the data receiver using multi-threaded concurrent execution;

[0011] Step 4: The data receiver combines the segmented data blocks according to the preset decryption rules.

[0012] As a further aspect of the present invention: in step one, the specific process of segmenting and slicing the data to be pushed is as follows:

[0013] Mark the amount of data to be pushed as A, mark the preset data synchronization time as B, and mark the preset average transmission speed of a single thread during the data synchronization process as C.

[0014] Calculate the preset required number of threads D 初始 = (A / B) / C, and for D 初始 Round up to the nearest integer.

[0015] D 初始 The number of threads is compared with the preset value d, and the result determines whether to adjust the number of threads. The final number of threads D is then output. 最终 ;

[0016] Based on the final determined number of threads D 最终 The data to be pushed is divided into D segments of equal length. 最终 Each segmented data block.

[0017] As a further aspect of the present invention: the number of threads D 初始 The specific process for making the adjustment is as follows:

[0018] If D 初始 If <d, then the number of threads D is not specified. 初始 Make adjustments, D 最终 =D 初始 The data to be pushed will be segmented and sliced ​​into D. 初始 Each segmented data block;

[0019] If d≤D 初始 If the number of threads is less than (2*d), then adjust the number of threads to D. 最终 =(D 初始 +1), which means segmenting the data to be pushed into (D) 初始 +1) segmented data blocks;

[0020] If (2*d)≤D 初始 If the number of threads is less than (3*d), then adjust the number of threads to D. 最终 =(D 初始 +2), which means that the data to be pushed will be segmented and sliced ​​into (D) 初始 +2) segmented data blocks;

[0021] If D 初始 If the number of threads is ≥(3*d), then the number of threads will be adjusted to D. 最终 =(D 初始 +3), which means that the data to be pushed will be segmented and sliced ​​into (D) 初始 +3) segmented data blocks.

[0022] As a further aspect of the present invention, the encryption rule is specifically as follows:

[0023] D 最终 Each segmented data block is assigned a different code, and the code serves as the name of the segmented data block.

[0024] According to D 最终 The initial arrangement order of each segmented data block generates a list of code sorting combinations;

[0025] The code sorting and combination list is processed using an asymmetric encryption algorithm to generate a public key and a private key pair. The data sender uses the public key to encrypt the code sorting and combination list, and the data receiver uses the private key.

[0026] D 最终 Each segment of data block is randomly shuffled into an unordered state.

[0027] As a further aspect of the present invention: in step three, if an anomaly is detected during the segmented data block push process, a fault alarm is triggered. The specific process is as follows:

[0028] Construct a data transmission speed analysis sub-model, analyze the push process based on the data transmission speed analysis sub-model, and output the first analysis result;

[0029] Construct a time difference analysis sub-model, analyze the push process based on the time difference analysis sub-model, and output a second analysis result;

[0030] Based on the combined results of the first and second analyses, a fault warning of the corresponding level is issued.

[0031] As a further aspect of the present invention, the specific analysis process of the data transmission speed analysis sub-model is as follows:

[0032] Let Ei be the real-time data transmission speed of each thread, i = 1...D. 最终 ;

[0033] Filter out the maximum value in Ei and mark it as Emax; then filter out the minimum value in Ei and mark it as Emin.

[0034] Calculate the difference weight value V1 = (Emax - Emin) for the current data transmission speed;

[0035] Compare the difference weight value V1 with the preset value v1. If V1 > v1, output the first analysis result as abnormal data transmission speed; otherwise, proceed to the next step.

[0036] Each Ei is compared with the preset value e. If Ei > e, then Ei is assigned to the first sequence; if Ei ≤ e, then Ei is assigned to the second sequence.

[0037] Count the number of Ei in the first sequence and label it as X; count the number of Ei in the second sequence and label it as Y.

[0038] The thread anomaly level Z = Y / (X+Y) is calculated to determine the current data transmission speed.

[0039] The thread anomaly level Z is compared with the preset value z. If Z > z, the first analysis result is output as abnormal data transmission speed. If Z ≤ z, the first analysis result is output as normal data transmission speed.

[0040] As a further aspect of the present invention: the specific analysis process of the time difference analysis sub-model is as follows:

[0041] Let Fi be the time taken for each thread to complete data transmission, i = 1...D. 最终 ;

[0042] Filter out the maximum value in Fi and mark it as Fmax; then filter out the minimum value in Fi and mark it as Fmin.

[0043] Calculate the difference weight value V2 = (Fmax - Fmin) for the current data transmission time;

[0044] Compare the difference weight value V2 with the preset value v2. If V2 > v2, output the second analysis result as abnormal data transmission time; otherwise, proceed to the next step.

[0045] Each Fi is compared with a preset value F. If Fi > F, then Fi is assigned to the third sequence; if Fi ≤ F, then Fi is assigned to the fourth sequence.

[0046] Count the number of Fi in the third sequence and label it as W; count the number of Fi in the fourth sequence and label it as M.

[0047] Calculate the thread anomaly level Q = W / (W+M) for the current data transmission time.

[0048] The thread anomaly level Q is compared with a preset value q. If Q > q, the second analysis result is output as abnormal data transmission time; if Q ≤ q, the second analysis result is output as normal data transmission time.

[0049] As a further aspect of the present invention: the specific process of issuing a fault warning of the corresponding level by combining the first analysis result and the second analysis result is as follows:

[0050] If the first analysis result indicates abnormal data transmission speed, and the second analysis result indicates abnormal data transmission time, then the fault warning level issued is level three.

[0051] If the first analysis result indicates that the data transmission speed is normal, and the second analysis result indicates that the data transmission time is abnormal, then the fault warning level issued is level two.

[0052] If the first analysis result indicates abnormal data transmission speed, and the second analysis result indicates normal data transmission time, then the fault warning level issued is Level 1.

[0053] As a further aspect of the present invention: in step four, the decryption rule is specifically as follows:

[0054] The data recipient uses a private key to decrypt the list of sorted code combinations;

[0055] The unordered segmented data blocks are recombined by sorting and combining the decrypted code into a list.

[0056] The combined segmented data blocks are output as complete data.

[0057] Compared with the prior art, the beneficial effects of the present invention are:

[0058] 1. This application pushes the data to be pushed to the data receiver by segmenting and slicing the data, and then using multiple threads to run concurrently. Finally, the data receiver combines the segmented data blocks to achieve the goal of efficient data synchronization.

[0059] 2. When segmenting and slicing data, this application selects an appropriate number of threads to avoid causing other problems due to too many threads, while ensuring data synchronization time.

[0060] 3. This application monitors the segmented data blocks during the push process and issues graded fault alarms when anomalies are detected, thereby promptly reminding staff. Attached Figure Description

[0061] Figure 1 This is a flowchart of a method for efficiently pushing data during the synchronization of massive amounts of data. Detailed Implementation

[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] As mentioned in the background section of this application, research has found that existing data synchronization processes require data push. This is fine for small amounts of data, but for massive data synchronization tasks, data push will encounter considerable difficulties. The most common problem is the limitation of transmission speed, which has certain shortcomings.

[0064] To address the aforementioned shortcomings, this application discloses a method for efficiently pushing data during massive data synchronization. The method involves segmenting and slicing the data to be pushed, running it concurrently in multiple threads, and then having the data receiver combine the segmented data blocks to achieve efficient data synchronization.

[0065] The following will describe in detail, with reference to the accompanying drawings, how the solution of this application solves the above-mentioned technical problems.

[0066] Please see Figure 1 This invention provides a method for efficiently pushing data during massive data synchronization, comprising the following steps: Step 1: The data sender segments the data to be pushed into data blocks; Step 2: The data blocks are shuffled according to a preset encryption rule; Step 3: The data blocks are pushed to the data receiver via multi-threaded concurrent execution; Step 4: The data receiver reassembles the data blocks according to a preset decryption rule. This application uses segmentation to push the data to be pushed to the data receiver via multi-threaded concurrent execution, and finally the data receiver reassembles the data blocks, achieving efficient data synchronization.

[0067] In this embodiment, the specific process of segmenting and slicing the data to be pushed in step one is as follows: Mark the amount of data to be pushed as A, mark the preset data synchronization time as B, and mark the preset average transmission speed of a single thread during the data synchronization process as C; calculate the preset required number of threads D. 初始 = (A / B) / C, and for D 初始 Round up to the nearest integer; set D 初始 The number of threads is compared with the preset value d, and the result determines whether to adjust the number of threads. The final number of threads D is then output. 最终 Based on the final determined number of threads D 最终 The data to be pushed is divided into D segments of equal length. 最终 Each thread is segmented into data blocks. The creation, destruction, and context switching between threads consume system resources. When the number of threads is excessive, these overheads increase significantly, impacting overall system performance. Furthermore, in a multi-threaded environment, if multiple threads simultaneously access and modify shared data, it can lead to data inconsistency, deadlocks, and other problems. This application, when segmenting and slicing data, selects an appropriate number of threads while ensuring data synchronization time, avoiding problems caused by an excessive number of threads.

[0068] In this embodiment, the number of threads D 初始 The specific process for making the adjustment is as follows: If D 初始 If <d, then the number of threads D is not specified. 初始 Make adjustments, D 最终 =D 初始 The data to be pushed will be segmented and sliced ​​into D. 初始 A segmented data block; if d≤D 初始 If the number of threads is less than (2*d), then adjust the number of threads to D. 最终 =(D 初始 +1), which means segmenting the data to be pushed into (D) 初始 +1) segmented data blocks; if (2*d)≤D 初始 If the number of threads is less than (3*d), then adjust the number of threads to D. 最终 =(D 初始 +2), which means that the data to be pushed will be segmented and sliced ​​into (D) 初始 +2) segmented data blocks; if D 初始 If the number of threads is ≥(3*d), then the number of threads will be adjusted to D. 最终 =(D 初始 +3), which means that the data to be pushed will be segmented and sliced ​​into (D) 初始 +3) segmented data blocks. This setting is based on the currently preset initial thread count D. 初始 Adjustments should be made. The more initial threads there are, the heavier the workload of the synchronization task, and the more threads are needed to reduce the workload.

[0069] In this embodiment, the encryption rule is specifically: for D 最终 Each segmented data block is assigned a different code, and the code serves as the name of the segmented data block; according to D 最终 The initial arrangement of the segmented data blocks generates a code sorting combination list; this list is then processed using an asymmetric encryption algorithm to generate a public and private key pair (public and private keys are cryptographic keys commonly used for data encryption, decryption, signing, and verification; they are fundamental to modern network security and information security). The data sender uses the public key to encrypt the code sorting combination list, and the data receiver uses the private key; D 最终 Each segment of data is randomly shuffled into an unordered state. This setting further enhances the security of data transmission.

[0070] In this embodiment, in step three, if an anomaly is detected during the segmented data block push process, a fault alarm is triggered. Specifically, the process involves: constructing a data transmission speed analysis sub-model; analyzing the push process based on this sub-model and outputting a first analysis result; constructing a time difference analysis sub-model; analyzing the push process based on this sub-model and outputting a second analysis result; and combining the first and second analysis results to issue a fault warning of the appropriate level. This setting effectively monitors whether anomalies exist during the data push process.

[0071] In this embodiment, the specific analysis process of the data transmission speed analysis sub-model is as follows: the real-time data transmission speed of each thread is denoted as Ei, i = 1...D 最终 The process involves: filtering out the maximum value in Ei and marking it as Emax; filtering out the minimum value in Ei and marking it as Emin; calculating the difference weight value V1 = (Emax - Emin) for the current data transmission speed; comparing the difference weight value V1 with the preset value v1; if V1 > v1, outputting the first analysis result as an abnormal data transmission speed; otherwise, proceeding to the next step; comparing each Ei with the preset value e; if Ei > e, assigning Ei to the first sequence; if Ei ≤ e, assigning Ei to the second sequence; counting the number of Ei in the first sequence and marking it as X, and counting the number of Ei in the second sequence and marking it as Y; calculating the thread abnormality degree Z = Y / (X + Y) for the current data transmission speed; comparing the thread abnormality degree Z with the preset value z; if Z > z, outputting the first analysis result as an abnormal data transmission speed; if Z ≤ z, outputting the first analysis result as a normal data transmission speed. This setting can quickly determine whether there is an abnormality in the data transmission speed.

[0072] In this embodiment, the specific analysis process of the time difference analysis sub-model is as follows: the time consumed by each thread to complete data transmission is marked as Fi, i = 1...D 最终The process involves: filtering out the maximum value in Fi and marking it as Fmax; filtering out the minimum value in Fi and marking it as Fmin; calculating the difference weight value V2 = (Fmax - Fmin) for the current data transmission time; comparing the difference weight value V2 with the preset value v2; if V2 > v2, outputting the second analysis result as abnormal data transmission time; otherwise, proceeding to the next step; comparing each Fi with the preset value F; if Fi > F, allocating Fi to the third sequence; if Fi ≤ F, allocating Fi to the fourth sequence; counting the number of Fi in the third sequence and marking it as W, and counting the number of Fi in the fourth sequence and marking it as M; calculating the thread abnormality degree Q = W / (W + M) for the current data transmission time; comparing the thread abnormality degree Q with the preset value q; if Q > q, outputting the second analysis result as abnormal data transmission time; if Q ≤ q, outputting the second analysis result as normal data transmission time. This setting can quickly determine whether there is an abnormality in data transmission time.

[0073] In this embodiment, the specific process of issuing a fault warning of corresponding level based on the combined results of the first and second analyses is as follows: if the first analysis result indicates abnormal data transmission speed and the second analysis result indicates abnormal data transmission time, the issued fault warning level is level three; if the first analysis result indicates normal data transmission speed and the second analysis result indicates abnormal data transmission time, the issued fault warning level is level two; if the first analysis result indicates abnormal data transmission speed and the second analysis result indicates normal data transmission time, the issued fault warning level is level one. This setting allows staff to easily understand the severity and urgency of abnormalities during the current data transmission process.

[0074] In this embodiment, in step four, the decryption rule is as follows: the data receiver uses its private key to decrypt the code sorting and combination list; the unordered segmented data blocks are then recombined using the decrypted code sorting and combination list; and the recombined segmented data blocks are output as complete data. This setting improves the security of data transmission.

[0075] This invention uses a segmented data slicing method, with multiple threads running concurrently to push the data to the data receiver. Finally, the data receiver combines the segmented data blocks to achieve efficient data synchronization. When segmenting the data, this application selects an appropriate number of threads while ensuring data synchronization time, avoiding problems caused by excessive threads. This application monitors the data block push process and issues tiered fault alarms when anomalies are detected, thus promptly alerting staff.

[0076] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0077] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for efficiently pushing data during massive data synchronization, characterized in that, Includes the following steps: Step 1: The data sender segments the data to be pushed into data blocks. Step 2: Shuffle the segmented data blocks according to the preset encryption rules; Step 3: Push segmented data blocks to the data receiver using multi-threaded concurrent execution; Step 4: The data receiver combines the segmented data blocks according to the preset decryption rules; In step one, the specific process of segmenting and slicing the data to be pushed is as follows: Mark the amount of data to be pushed as A, mark the preset data synchronization time as B, and mark the preset average transmission speed of a single thread during the data synchronization process as C. Calculate the preset required number of threads D 初始 = (A / B) / C, and for D 初始 Round up to the nearest integer. D 初始 The number of threads is compared with the preset value d, and the result determines whether to adjust the number of threads. The final number of threads D is then output. 最终 ; Based on the final determined number of threads D 最终 The data to be pushed is divided into D segments of equal length. 最终 Each segmented data block; The number of threads D 初始 The specific process for making the adjustment is as follows: If D 初始 If <d, then the number of threads D is not specified. 初始 Make adjustments, D 最终 =D 初始 The data to be pushed will be segmented and sliced ​​into D. 初始 Each segmented data block; If d≤D 初始 If the number of threads is less than (2*d), then adjust the number of threads to D. 最终 =(D 初始 +1), which means that the data to be pushed will be segmented into (D) 初始 +1) segmented data blocks; If (2*d)≤D 初始 If the number of threads is less than (3*d), then adjust the number of threads to D. 最终 =(D 初始 +2), which means that the data to be pushed will be segmented into (D) 初始 +2) segmented data blocks; If D 初始 If the number of threads is greater than or equal to (3*d), then the number of threads will be adjusted to D. 最终 =(D 初始 +3), which means that the data to be pushed will be segmented and sliced ​​into (D) 初始 +3) segmented data blocks.

2. The method for efficiently pushing data during massive data synchronization according to claim 1, characterized in that, The encryption rules are as follows: D 最终 Each segmented data block is assigned a different code, and the code serves as the name of the segmented data block. According to D 最终 The initial arrangement order of each segmented data block generates a list of code sorting combinations; The code sorting and combination list is processed using an asymmetric encryption algorithm to generate a public key and a private key pair. The data sender uses the public key to encrypt the code sorting and combination list, and the data receiver uses the private key. D 最终 Each segment of data block is randomly shuffled into an unordered state.

3. The method for efficiently pushing data during massive data synchronization according to claim 2, characterized in that, In step three, if an anomaly is detected during the segmented data block push process, a fault alarm will be triggered. The specific process is as follows: Construct a data transmission speed analysis sub-model, analyze the push process based on the data transmission speed analysis sub-model, and output the first analysis result; Construct a time difference analysis sub-model, analyze the push process based on the time difference analysis sub-model, and output a second analysis result; Based on the combined results of the first and second analyses, a fault warning of the corresponding level is issued.

4. The method for efficiently pushing data during massive data synchronization according to claim 3, characterized in that, The specific analysis process of the data transmission speed analysis sub-model is as follows: Let Ei be the real-time data transmission speed of each thread, i=1...D 最终 ; Filter out the maximum value in Ei and mark it as Emax; then filter out the minimum value in Ei and mark it as Emin. Calculate the difference weight value of the current data transmission speed, V1 = (Emax - Emin); Compare the difference weight value V1 with the preset value v1. If V1 > v1, output the first analysis result as abnormal data transmission speed; otherwise, proceed to the next step. Each Ei is compared with the preset value e. If Ei > e, then Ei is assigned to the first sequence; if Ei ≤ e, then Ei is assigned to the second sequence. Count the number of Ei in the first sequence and label it as X; count the number of Ei in the second sequence and label it as Y. The thread anomaly level Z is calculated as Z = Y / (X+Y) to determine the current data transmission speed. The thread anomaly level Z is compared with the preset value z. If Z > z, the first analysis result is output as abnormal data transmission speed. If Z ≤ z, the first analysis result is output as normal data transmission speed.

5. A method for efficiently pushing data during massive data synchronization as described in claim 4, characterized in that, The specific analysis process of the time difference analysis sub-model is as follows: Let Fi be the time taken for each thread to complete data transmission, i=1...D. 最终 ; Filter out the maximum value in Fi and mark it as Fmax; then filter out the minimum value in Fi and mark it as Fmin. Calculate the difference weight value V2 = (Fmax - Fmin) for the current data transmission time. Compare the difference weight value V2 with the preset value v2. If V2 > v2, output the second analysis result as abnormal data transmission time; otherwise, proceed to the next step. Each Fi is compared with a preset value F. If Fi > F, then Fi is assigned to the third sequence; if Fi ≤ F, then Fi is assigned to the fourth sequence. Count the number of Fi in the third sequence and label it as W; count the number of Fi in the fourth sequence and label it as M. The thread anomaly level Q = W / (W+M) is calculated to determine the current data transmission time consumption. The thread anomaly level Q is compared with a preset value q. If Q > q, the second analysis result is output as abnormal data transmission time; if Q ≤ q, the second analysis result is output as normal data transmission time.

6. A method for efficiently pushing data during massive data synchronization as described in claim 5, characterized in that, The specific process for issuing a fault warning of the corresponding level by combining the results of the first and second analyses is as follows: If the first analysis result indicates abnormal data transmission speed, and the second analysis result indicates abnormal data transmission time, then the fault warning level issued is level three. If the first analysis result indicates that the data transmission speed is normal, and the second analysis result indicates that the data transmission time is abnormal, then the fault warning level issued is level two. If the first analysis result indicates abnormal data transmission speed, and the second analysis result indicates normal data transmission time, then the fault warning level issued is Level 1.

7. A method for efficiently pushing data during massive data synchronization as described in claim 6, characterized in that, In step four, the decryption rules are as follows: The data recipient uses a private key to decrypt the list of sorted code combinations; The unordered segmented data blocks are recombined by sorting and combining the decrypted code into a list. The combined segmented data blocks are output as complete data.

Images