Method for efficiently pushing data in mass data synchronization process

Abstract

The invention discloses a method for efficiently pushing data in a mass data synchronization process, which belongs to the field of data pushing methods, and comprises the following steps of: 1, segmenting and slicing to-be-pushed data by a data sender to form segmented data blocks; 2, disorganizing the segmented data blocks according to a preset encryption rule; 3, pushing the segmented data blocks to a data receiver through multi-thread concurrent operation; and 4, the data receiver combines the segmented data blocks according to a preset decryption rule. According to the method, the to-be-pushed data is pushed to the data receiving party through the segmentation and slicing means and multi-thread concurrent operation, and finally, the data receiving party combines the segmented data blocks, so that the purpose of efficiently synchronizing the data is achieved.

Description

Technical Field

[0001] The present invention relates to the field of data push methods, and in particular to a method for efficiently pushing data during mass data synchronization. Background Art

[0002] Data synchronization refers to the real-time or regular updating and matching of data in two or more different locations, systems or devices to maintain data consistency and integrity. Data synchronization can improve data reliability, security, availability and sharing, providing strong support for business decision-making and data analysis.

[0003] During the data synchronization process, data needs to be pushed. If the amount of data is small, it is fine. However, if the synchronization task is massive data, there will be considerable difficulties in data push. The most common difficulty is the restriction of transmission speed. Due to the huge amount of data, transmission speed often becomes the key factor restricting synchronization efficiency. Network bandwidth, storage device read and write speed, and the quality of synchronization algorithm will directly affect the transmission speed. Therefore, how to improve the data push speed during the massive data synchronization process has become a problem that needs to be solved urgently.

[0004] Therefore, those skilled in the art provide a method for efficiently pushing data during the synchronization of massive data to solve the problems raised in the above background technology. Summary of the invention

[0005] The purpose of the present invention is to provide a method for efficiently pushing data during the synchronization of massive data, wherein the data to be pushed is pushed to a data receiver through segmentation and slicing, and multi-threaded concurrent operation. Finally, the data receiver combines the segmented data blocks to achieve the purpose of efficient data synchronization, so as to solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A method for efficiently pushing data during mass data synchronization includes the following steps:

[0008] Step 1: The data sender segments and slices the data to be pushed to form segmented data blocks;

[0009] Step 2: Scramble the segmented data blocks according to the preset encryption rules;

[0010] Step 3: Push the segmented data blocks to the data receiver through multi-threaded concurrent operation;

[0011] Step 4: The data receiver combines the segmented data blocks according to the preset decryption rules.

[0012] As a further solution of the present invention: in step 1, the specific process of segmenting and slicing the data to be pushed is:

[0013] The amount of data to be pushed is marked as A, the preset data synchronization time is marked as B, and the preset average transmission speed of a single thread during the data synchronization process is marked as C;

[0014] Calculate the preset number of threads D required 初始 =(A / B) / C, and for D 初始 Use the round-up method to round the integer;

[0015] D 初始 Compare with the preset value d, decide whether to adjust the number of threads based on the comparison result, and output the final number of threads D 最终 ;

[0016] According to the final number of threads D 最终 Divide the data to be pushed into D segments evenly 最终 Segmented data blocks.

[0017] As a further solution of the present invention: the number of threads D 初始 The specific process of adjustment is:

[0018] If D 初始 <d, then the number of threads D is not correct 初始 Make adjustments, D 最终 =D 初始 , that is, the data to be pushed is segmented into D 初始 Segmented data blocks;

[0019] If d≤D 初始 <(2*d), then adjust the number of threads to D 最终 =(D 初始 +1), that is, the data to be pushed is segmented into (D 初始 +1) segment data blocks;

[0020] If (2*d)≤D 初始 <(3*d), then adjust the number of threads to D 最终 =(D 初始 +2), that is, the data to be pushed is segmented into (D 初始 +2) segmented data blocks;

[0021] If D 初始 ≥(3*d), then adjust the number of threads to D 最终 =(D 初始 +3), which means to divide the data to be pushed into segments (D 初始 +3) fragment data blocks.

[0022] As a further solution of the present invention: the encryption rule is specifically:

[0023] D 最终 Each segmented data block is assigned a different code, and the code serves as the name of the segmented data block;

[0024] According to D 最终 The initial arrangement order of the segmented data blocks generates a code sorting combination list;

[0025] The code sorting combination list is encrypted by an asymmetric algorithm to generate a pair of public and private keys. The data sender uses the public key to encrypt the code sorting combination list, and the data receiver uses the private key.

[0026] D 最终 The segmented data blocks are randomly shuffled into a disordered state.

[0027] As a further solution of the present invention: in the step 3, if an abnormality is detected during the push of the segmented data blocks, a fault alarm is issued, and the specific process is as follows:

[0028] Constructing a data transmission speed analysis sub-model, analyzing the push process based on the data transmission speed analysis sub-model, and outputting a first analysis result;

[0029] Constructing a time difference analysis sub-model, analyzing the push process based on the time difference analysis sub-model, and outputting a second analysis result;

[0030] Based on the first analysis result and the second analysis result, a fault warning of a corresponding level is issued.

[0031] As a further solution of the present invention: the specific analysis process of the data transmission speed analysis sub-model is as follows:

[0032] The real-time data transmission speed of each thread is marked as Ei, i=1···D 最终 ;

[0033] Filter out the maximum value in Ei and mark it as Emax, and then filter out the minimum value in Ei and mark it as Emin;

[0034] Calculate the difference weight value V1 of the current data transmission speed = (Emax-Emin);

[0035] Compare the difference weight value V1 with the preset value v1, if V1>v1, output the first analysis result that the data transmission speed is abnormal, otherwise proceed to the next step;

[0036] Compare each Ei with the preset value e. If Ei>e, Ei is planned to the first sequence. If Ei≤e, Ei is planned to the second sequence.

[0037] Count the number of Ei in the first sequence and mark it as X, and count the number of Ei in the second sequence and mark it as Y;

[0038] Calculate the thread abnormality level Z=Y / (X+Y) of the current data transmission speed;

[0039] The thread abnormality degree Z is compared with the preset value z. If Z>z, the first analysis result is output as abnormal data transmission speed. If Z≤z, the first analysis result is output as normal data transmission speed.

[0040] As a further solution of the present invention: the specific analysis process of the time difference analysis sub-model is:

[0041] The time taken by each thread to complete data transmission is marked as Fi, i = 1...D 最终 ;

[0042] Filter out the maximum value in Fi and mark it as Fmax, and then filter out the minimum value in Fi and mark it as Fmin;

[0043] Calculate the difference weight value V2 of the current data transmission time consumption = (Fmax-Fmin);

[0044] Compare the difference weight value V2 with the preset value v2. If V2>v2, output the second analysis result that the data transmission time is abnormal, otherwise proceed to the next step;

[0045] Compare each Fi with the preset value F. If Fi>F, Fi is planned to the third sequence. If Fi≤F, Fi is planned to the fourth sequence.

[0046] Count the number of Fi in the third sequence and mark it as W, and count the number of Fi in the fourth sequence and mark it as M;

[0047] Calculate the abnormal degree of the thread Q = W / (W+M) for the current data transmission time;

[0048] The thread abnormality degree Q is compared with the preset value q. If Q>q, the second analysis result is output as abnormal data transmission time. If Q≤q, the second analysis result is output as normal data transmission time.

[0049] As a further solution of the present invention: the specific process of issuing a corresponding level of fault warning by integrating the first analysis result and the second analysis result is:

[0050] If the first analysis result is that the data transmission speed is abnormal, and the second analysis result is that the data transmission time is abnormal, the fault warning level issued is level three;

[0051] If the first analysis result shows that the data transmission speed is normal and the second analysis result shows that the data transmission time is abnormal, the fault warning level issued is level 2;

[0052] If the first analysis result is that the data transmission speed is abnormal and the second analysis result is that the data transmission time is normal, the fault warning level issued is level one.

[0053] As a further solution of the present invention: in step 4, the decryption rule is specifically:

[0054] The data receiver uses the private key to decrypt the code sorting combination list;

[0055] Reassemble the unordered segmented data blocks through the decrypted code sort combination list;

[0056] The combined segmented data blocks are output as complete data.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1. This application pushes the data to be pushed to the data receiver by means of segmentation and slicing, and multi-threaded concurrent operation. Finally, the data receiver combines the segmented data blocks to achieve the purpose of efficient data synchronization.

[0059] 2. When segmenting and slicing data, this application tries to select an appropriate number of threads as much as possible while ensuring data synchronization time to avoid other problems caused by too many threads.

[0060] 3. This application monitors the segmented data blocks during their push process, and issues graded fault alarms when abnormalities are detected, thereby promptly alerting staff. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 The present invention is a flowchart of a method for efficiently pushing data during the synchronization of massive data. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] As mentioned in the background technology of this application, research has found that in the existing data synchronization process, data needs to be pushed. If the amount of data is small, it is fine. However, if a massive amount of data is required for synchronization, then there will be considerable difficulties in data pushing. The most common difficulty is the limitation of transmission speed, which has certain defects.

[0064] In order to solve the above-mentioned defects, the present application discloses a method for efficiently pushing data during the synchronization of massive data, in which the data to be pushed is pushed to the data receiver through segmentation and slicing, and multi-threaded concurrent operation. Finally, the data receiver combines the segmented data blocks to achieve the purpose of efficient data synchronization.

[0065] The following will describe in detail how the solution of the present application solves the above technical problems in conjunction with the accompanying drawings.

[0066] See also Figure 1 In an embodiment of the present invention, a method for efficiently pushing data during the synchronization of massive data includes the following steps: Step 1: The data sender segments and slices the data to be pushed to form segmented data blocks; Step 2: The segmented data blocks are shuffled according to the preset encryption rules; Step 3: The segmented data blocks are pushed to the data receiver through multi-threaded concurrent operation; Step 4: The data receiver combines the segmented data blocks according to the preset decryption rules. This application pushes the data to be pushed to the data receiver through segmented slicing and multi-threaded concurrent operation, and finally the data receiver combines the segmented data blocks to achieve the purpose of efficient data synchronization.

[0067] In this embodiment, in step 1, the specific process of segmenting the data to be pushed is: marking the amount of data to be pushed as A, marking the preset data synchronization time as B, and marking the preset average transmission speed of a single thread during the data synchronization process as C; calculating the preset number of threads required D 初始 =(A / B) / C, and for D 初始 Use the round-up method to round the integer; D 初始 Compare with the preset value d, decide whether to adjust the number of threads based on the comparison result, and output the final number of threads D 最终 ; According to the final number of threads D 最终 Divide the data to be pushed into D segments evenly 最终 Segmented data blocks. The creation and destruction of each thread and the context switching between threads all consume system resources. When the number of threads is too large, these overheads will increase significantly, affecting the overall performance of the system. In addition, in a multi-threaded environment, if multiple threads access and modify shared data at the same time, it may cause data inconsistency, deadlock and other problems. When slicing data, this application tries to select an appropriate number of threads as much as possible while ensuring data synchronization time to avoid other problems caused by too many threads.

[0068] In this embodiment, the number of threads D 初始 The specific process of adjustment is: if D 初始 <d, then the number of threads D is not correct 初始 Make adjustments, D 最终 =D 初始 , that is, the data to be pushed is segmented into D 初始 segmented data blocks; if d≤D 初始 <(2*d), then adjust the number of threads to D 最终 =(D 初始 +1), that is, the data to be pushed is segmented into (D 初始 +1) segmented data blocks; if (2*d)≤D 初始 <(3*d), then adjust the number of threads to D 最终 =(D 初始 +2), that is, the data to be pushed is segmented into (D 初始 +2) segmented data blocks; if D 初始 ≥(3*d), then adjust the number of threads to D 最终 =(D 初始 +3), which means to divide the data to be pushed into segments (D 初始 +3) segment data blocks. This setting is based on the current preset initial number of threads D 初始 Make adjustments. The more initial threads there are, the heavier the workload of the synchronization task is, and the more threads need to be budgeted to reduce the workload.

[0069] In this embodiment, the encryption rule is as follows: 最终 Each segmented data block is assigned a different code, and the code is used as the name of the segmented data block; according to D 最终 The initial arrangement order of the segmented data blocks generates a code sorting combination list; the code sorting combination list is encrypted through an asymmetric encryption algorithm to generate a pair of public and private keys (public and private keys are a pair of keys in cryptography, usually used to implement data encryption, decryption, signing and verification operations. They are the basis of modern network security and information security). The data sender uses the public key to encrypt the code sorting combination list, and the data receiver uses the private key; D 最终 The segmented data blocks are randomly shuffled into a disordered state. This setting can further improve the security of the data transmission process.

[0070] In this embodiment, in step 3, if an abnormality is detected during the push of the segmented data blocks, a fault alarm is issued. The specific process is: construct a data transmission speed analysis sub-model, analyze the push process based on the data transmission speed analysis sub-model, and output a first analysis result; construct a time difference analysis sub-model, analyze the push process based on the time difference analysis sub-model, and output a second analysis result; combine the first analysis result and the second analysis result to issue a corresponding level of fault warning. This setting can effectively monitor whether there is an abnormality in the data push process.

[0071] In this embodiment, the specific analysis process of the data transmission speed analysis sub-model is as follows: mark the real-time data transmission speed of each thread as Ei, i=1...D 最终 ; Filter out the maximum value in Ei and mark it as Emax, and then filter out the minimum value in Ei and mark it as Emin; calculate the difference weight value V1 = (Emax-Emin) of the current data transmission speed; compare the difference weight value V1 with the preset value v1, if V1>v1, then output the first analysis result as abnormal data transmission speed, otherwise proceed to the next step; compare each Ei with the preset value e, if Ei>e, then plan Ei to the first sequence, if Ei≤e, then plan Ei to the second sequence; count the number of Ei in the first sequence and mark it as X, count the number of Ei in the second sequence and mark it as Y; calculate the thread abnormality degree Z = Y / (X+Y) of the current data transmission speed; compare the thread abnormality degree Z with the preset value z, if Z>z, then output the first analysis result as abnormal data transmission speed, if Z≤z, then output the first analysis result as normal data transmission speed. This setting can quickly determine whether the data transmission speed is abnormal.

[0072] In this embodiment, the specific analysis process of the time difference analysis sub-model is as follows: the time taken to complete the data transmission of each thread is marked as Fi, i=1...D 最终; Filter out the maximum value in Fi and mark it as Fmax, and then filter out the minimum value in Fi and mark it as Fmin; calculate the difference weight value V2 = (Fmax-Fmin) of the current data transmission time; compare the difference weight value V2 with the preset value v2, if V2>v2, then output the second analysis result as abnormal data transmission time, otherwise proceed to the next step; compare each Fi with the preset value F, if Fi>F, then plan Fi to the third sequence, if Fi≤F, then plan Fi to the fourth sequence; count the number of Fi in the third sequence and mark it as W, count the number of Fi in the fourth sequence and mark it as M; calculate the thread abnormality degree Q = W / (W+M) of the current data transmission time; compare the thread abnormality degree Q with the preset value q, if Q>q, then output the second analysis result as abnormal data transmission time, if Q≤q, then output the second analysis result as normal data transmission time. This setting can quickly determine whether there is an abnormality in the data transmission time.

[0073] In this embodiment, the specific process of issuing a corresponding level of fault warning based on the first analysis result and the second analysis result is as follows: if the first analysis result is that the data transmission speed is abnormal and the second analysis result is that the data transmission time is abnormal, the fault warning level issued is level 3; if the first analysis result is that the data transmission speed is normal and the second analysis result is that the data transmission time is abnormal, the fault warning level issued is level 2; if the first analysis result is that the data transmission speed is abnormal and the second analysis result is that the data transmission time is normal, the fault warning level issued is level 1. This setting can facilitate the staff to understand the severity of the abnormality in the current data transmission process.

[0074] In this embodiment, in step 4, the decryption rule is specifically: the data receiver uses the private key to decrypt the code sorting combination list; reassembles the disordered segmented data blocks through the decrypted code sorting combination list; and outputs the reassembled segmented data blocks as complete data. This setting can improve the security of data transmission.

[0075] The present invention pushes the data to be pushed to the data receiver through segmentation and slicing, and multi-threaded concurrent operation. Finally, the data receiver combines the segmented data blocks to achieve the purpose of efficient data synchronization. When the application segments and slices the data, it selects the appropriate number of threads as much as possible while ensuring the data synchronization time, so as to avoid other problems caused by too many threads. The application monitors the segmented data blocks during their pushing process, and issues a graded fault alarm when an abnormality is detected, thereby promptly reminding the staff.

[0076] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0077] What is described above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for efficiently pushing data during mass data synchronization, characterized in that: The following steps are involved: Step 1: The data sender segments and slices the data to be pushed to form segmented data blocks; Step 2: Scramble the segmented data blocks according to the preset encryption rules; Step 3: Push the segmented data blocks to the data receiver through multi-threaded concurrent operation; Step 4: The data receiver combines the segmented data blocks according to the preset decryption rules.

2. A method for efficiently pushing data in a massive data synchronization process according to claim 1, characterized in that: In step 1, the specific process of segmenting the data to be pushed is as follows: The amount of data to be pushed is marked as A, the preset data synchronization time is marked as B, and the preset average transmission speed of a single thread during the data synchronization process is marked as C; Calculate the preset number of threads D required 初始 =(A / B) / C, and for D 初始 Use the round-up method to round the integer; D 初始 Compare with the preset value d, decide whether to adjust the number of threads based on the comparison result, and output the final number of threads D 最终 ; According to the final number of threads D 最终 Divide the data to be pushed into D segments evenly 最终 Segmented data blocks.

3. A method for efficiently pushing data in a massive data synchronization process according to claim 2, characterized in that: The number of threads D 初始 The specific process of adjustment is as follows: If D 初始 <d, then the number of threads D is not correct 初始 Make adjustments, D 最终 =D 初始 , that is, the data to be pushed is segmented into D 初始 Segmented data blocks; If d≤D 初始 <(2*d), then adjust the number of threads to D 最终 =(D 初始 +1), that is, the data to be pushed is segmented into (D 初始 +1) segment data blocks; If (2*d)≤D 初始 <(3*d), then adjust the number of threads to D 最终 =(D 初始 +2), that is, the data to be pushed is segmented into (D 初始 +2) segmented data blocks; If D 初始 ≥(3*d), then adjust the number of threads to D 最终 =(D 初始 +3), which means to divide the data to be pushed into segments (D 初始 +3) fragment data blocks.

4. A method for efficiently pushing data in a massive data synchronization process according to claim 3, characterized in that: The encryption rules are specifically as follows: D 最终 Each segmented data block is assigned a different code, and the code serves as the name of the segmented data block; According to D 最终 The initial arrangement order of the segmented data blocks generates a code sorting combination list; The code sorting combination list is encrypted by an asymmetric algorithm to generate a pair of public and private keys. The data sender uses the public key to encrypt the code sorting combination list, and the data receiver uses the private key. D 最终 The segmented data blocks are randomly shuffled into a disordered state.

5. A method for efficiently pushing data in a massive data synchronization process according to claim 4, characterized in that: In step 3, if an abnormality is detected during the push of the segmented data blocks, a fault alarm is issued. The specific process is as follows: Constructing a data transmission speed analysis sub-model, analyzing the push process based on the data transmission speed analysis sub-model, and outputting a first analysis result; Constructing a time difference analysis sub-model, analyzing the push process based on the time difference analysis sub-model, and outputting a second analysis result; Based on the first analysis result and the second analysis result, a fault warning of a corresponding level is issued.

6. A method for efficiently pushing data in a massive data synchronization process according to claim 5, characterized in that: The specific analysis process of the data transmission speed analysis sub-model is as follows: The real-time data transmission speed of each thread is marked as Ei, i=1···D 最终 ; Filter out the maximum value in Ei and mark it as Emax, and then filter out the minimum value in Ei and mark it as Emin; Calculate the difference weight value V1 of the current data transmission speed = (Emax-Emin); Compare the difference weight value V1 with the preset value v1, if V1>v1, output the first analysis result that the data transmission speed is abnormal, otherwise proceed to the next step; Compare each Ei with the preset value e. If Ei>e, Ei is planned to the first sequence. If Ei≤e, Ei is planned to the second sequence. Count the number of Ei in the first sequence and mark it as X, and count the number of Ei in the second sequence and mark it as Y; Calculate the thread abnormality level Z=Y / (X+Y) of the current data transmission speed; The thread abnormality degree Z is compared with the preset value z. If Z>z, the first analysis result is output as abnormal data transmission speed. If Z≤z, the first analysis result is output as normal data transmission speed.

7. A method for efficiently pushing data in a massive data synchronization process according to claim 6, characterized in that: The specific analysis process of the time difference analysis sub-model is as follows: The time taken by each thread to complete data transmission is marked as Fi, i = 1...D 最终 ; Filter out the maximum value in Fi and mark it as Fmax, and then filter out the minimum value in Fi and mark it as Fmin; Calculate the difference weight value V2 of the current data transmission time consumption = (Fmax-Fmin); Compare the difference weight value V2 with the preset value v2. If V2>v2, output the second analysis result that the data transmission time is abnormal, otherwise proceed to the next step; Compare each Fi with the preset value F. If Fi>F, Fi is planned to the third sequence. If Fi≤F, Fi is planned to the fourth sequence. Count the number of Fi in the third sequence and mark it as W, and count the number of Fi in the fourth sequence and mark it as M; Calculate the abnormality level of the thread that consumes the current data transmission time Q = W / (W+M); The thread abnormality degree Q is compared with the preset value q. If Q>q, the second analysis result is output as abnormal data transmission time. If Q≤q, the second analysis result is output as normal data transmission time.

8. A method for efficiently pushing data in a massive data synchronization process according to claim 7, characterized in that: The specific process of issuing a corresponding level of fault warning by combining the first analysis result and the second analysis result is as follows: If the first analysis result is that the data transmission speed is abnormal, and the second analysis result is that the data transmission time is abnormal, the fault warning level issued is level three; If the first analysis result shows that the data transmission speed is normal and the second analysis result shows that the data transmission time is abnormal, the fault warning level issued is level 2; If the first analysis result is that the data transmission speed is abnormal and the second analysis result is that the data transmission time is normal, the fault warning level issued is level one.

9. A method for efficiently pushing data in a massive data synchronization process according to claim 8, characterized in that: In step 4, the decryption rules are specifically as follows: The data receiver uses the private key to decrypt the code sorting combination list; Reassemble the unordered segmented data blocks through the decrypted code sort combination list; The combined segmented data blocks are output as complete data.

Images