A method for realizing breakpoint-resume transmission based on offline synchronization

Abstract

The present invention relates to the field of data processing technology, and in particular to a method for realizing breakpoint-resume transmission based on offline synchronization, comprising the following steps: slicing an original file to output a number of slicing data, and transmitting the several slicing data offline to a target location when a network is interrupted; transmitting the slicing data according to the breakpoints of the slicing data when the network is interrupted, and merging the slicing data transmitted to the target location in the order of slicing to restore the original file; judging the stability of breakpoint-resume transmission based on the variance of the proportion of lost bytes of the slicing data; if it is determined that the stability does not meet the requirements, adjusting the byte amount of the slicing overlapping area; if it is determined that the reliability does not meet the requirements, adjusting the transmission rate of the slicing data for the first time; and adjusting the transmission rate of the slicing data for the second time based on the average recovery time of the slicing data. The present invention achieves an improvement in the stability of breakpoint-resume transmission.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for implementing breakpoint-resume transmission based on offline synchronization. Background Art

[0002] In the existing technology, the core of the breakpoint resume mechanism is to be able to identify and record the progress status of data synchronization, ensuring that after the task is interrupted for some reason, it can be restored from the precise interruption position instead of starting from the beginning. This is especially important for processing long-running tasks, because it is directly related to the effective use of resources, the timeliness of tasks, and the overall reliability of the system. For example, the amount of data synchronized in the entire database is often large and needs to run for several hours or even days. It cannot be synchronized from the beginning due to network jitter, individual table synchronization failure, etc. It saves time and computing resources to a great extent.

[0003] The implementation method of breakpoint resume is based on the checkpoint mechanism of the Flink computing engine (Flink checkpoint is an automatic fault-tolerant recovery mechanism, that is, a mechanism that saves the data synchronization status according to the specified time. The specified time can be set to the second or minute level, and the checkpoint data storage can be specified on the distributed file system. This mechanism ensures that even if an exception suddenly occurs during synchronization, it can be restored from the most recent or specified checkpoint). Breakpoint resume uses the checkpoint timing storage mechanism to store the status data of the synchronization process in the distributed file system. When the task fails and recovers, the location information of the checkpoint status data storage of the most recent successful synchronization is read, and the source data is read from the specified location to achieve the goal of breakpoint resume.

[0004] China Patent Publication No.: CN117033342A discloses a method and device for data synchronization and full data breakpoint resuming, including: setting an initial loading state for a table to be loaded during initial loading; grouping the table to be loaded to obtain group information, and saving the group information to a source end and a target end respectively; establishing a connection between the source end and the target end, and loading the group data according to the group information; updating the group information according to the result of the group data loading, and updating the initial loading state of the table to be loaded according to the updated group information; when a failure occurs during the loading of the group data, determining whether to reload according to the loading state of the table to be loaded after the update, if so, determining the group to be reloaded according to the loading state of each group in the table to be reloaded. It can be seen from this that the method and device for data synchronization and full data breakpoint resuming has the problem that part of the data block is lost when the network is interrupted, resulting in the inability to achieve correct alignment of the fragment when aligning with the data block, resulting in inaccurate fragment boundary processing, and then resulting in the inability to perform appropriate fragmentation when fragmenting the original file, thereby causing the stability of breakpoint resuming to decrease. Summary of the invention

[0005] To this end, the present invention provides a method for implementing breakpoint resume based on offline synchronization, so as to overcome the problem in the prior art that due to the loss of part of the data block during network interruption, the shards cannot be correctly aligned with the data blocks, resulting in inaccurate shard boundary processing, and further resulting in the inability to perform appropriate sharding when sharding the original file, thereby causing the stability of breakpoint resume to decrease.

[0006] To achieve the above-mentioned purpose, the present invention provides a method for implementing breakpoint-resume transmission based on offline synchronization, comprising the following steps: fragmenting an original file to output a number of fragment data, and transmitting the number of fragment data offline to a target location when a network is interrupted; transmitting the fragment data according to the breakpoint of the fragment data when the network is interrupted, and merging the fragment data transmitted to the target location in the order of the fragments to restore the original file; obtaining the amount of lost bytes of the fragment data within a number of recovery cycles; judging the stability of breakpoint-resume transmission based on the variance of the proportion of the amount of lost bytes of the fragment data; if it is determined that the stability does not meet the requirements, adjusting the amount of bytes in the overlapping area of ​​the fragments or executing a process for determining the reliability of the fragment data resuming transmission; if it is determined that the reliability does not meet the requirements, initially adjusting the transmission rate of the fragment data or adjusting the life cycle of the fragment data based on the proportion of the number of successful resuming transmissions of the original file; obtaining the recovery time of the fragment data after a single cycle of transmission of the fragment data at the transmission rate of the fragment data after the initial adjustment; and secondarily adjusting the transmission rate of the fragment data based on the average recovery time of the fragment data.

[0007] Furthermore, the original file resuming method includes fragment resuming, data row-level resuming and message queue data resuming.

[0008] Furthermore, the step of adjusting the byte amount of the overlapping area of ​​the slices includes:

[0009] Based on the variance of the percentage of lost bytes of shard data in several recovery cycles, it is determined that the stability of breakpoint-resume transmission does not meet the requirements;

[0010] If the variance of the percentage of lost bytes of the shard data meets the first variance condition or the second variance condition, it is determined that the stability of breakpoint resume transmission does not meet the requirement;

[0011] If the variance of the percentage of lost bytes of the shard data only satisfies the second variance condition, increasing the amount of bytes in the overlapping area of ​​the shards;

[0012] Among them, the first variance condition is that the variance of the proportion of lost bytes of the shard data is greater than the preset first variance and less than or equal to the preset second variance; the second variance condition is that the variance of the proportion of lost bytes of the shard data is greater than the preset second variance.

[0013] Furthermore, the process of determining the reliability of the resumed transmission of the sharded data includes: when the variance of the proportion of lost bytes of the sharded data only satisfies the first variance condition, preliminarily determining that the reliability of the resumed transmission of the sharded data does not meet the requirements, and performing a secondary determination on the reliability of the resumed transmission of the sharded data based on the average delay time of the sharded data transmission.

[0014] Furthermore, the increase in the amount of bytes in the overlapping area of ​​the slices is determined by the difference between the variance of the proportion of the amount of lost bytes in the slice data and a preset second variance.

[0015] Furthermore, the step of initially adjusting the transmission rate of the sliced ​​data includes:

[0016] Obtain the delay time of shard data transmission in several recovery cycles, and calculate the average delay time of shard data transmission;

[0017] Compare the average delay duration of the segmented data transmission with the preset first delay duration and the preset second delay duration respectively;

[0018] If the average delay time of the segment data transmission is greater than the preset first delay time, it is determined that the reliability of the segment data retransmission does not meet the requirement;

[0019] If the average delay time for transmitting the segmented data is greater than the preset first delay time and less than or equal to the preset second delay time, the transmission rate of the segmented data is reduced;

[0020] If the average delay time of the shard data transmission is greater than the preset second delay time, it is preliminarily determined that the storage validity of the shard data does not meet the requirements, and a secondary determination is made on the storage validity of the shard data based on the proportion of the number of successful resumed transmissions of the original files;

[0021] The reduction range of the transmission rate of the sliced ​​data is determined by the difference between the average delay duration of the sliced ​​data transmission and the preset first delay duration.

[0022] Furthermore, the calculation formula for the average delay time of the slice data transmission is:

[0023]

[0024] Among them, Z is the average delay time of shard data transmission, X a is the delay duration of shard data transmission in the ath recovery cycle, n is the number of recovery cycles, and n is a natural number greater than or equal to 1.

[0025] Furthermore, the step of adjusting the life cycle of the shard data includes:

[0026] Secondarily determine the storage validity of the shard data based on the percentage of the number of successfully resumed original files;

[0027] If the proportion of the number of original files successfully resumed is greater than the preset proportion, it is determined that the storage validity of the shard data does not meet the requirements, and the life cycle of the shard data is reduced;

[0028] The reduction extent of the life cycle of the shard data is determined by the difference between the proportion of the number of successfully resumed original files and a preset proportion.

[0029] Furthermore, the step of secondarily adjusting the transmission rate of the sliced ​​data includes:

[0030] Obtain the recovery time of shard data in several recovery cycles, and calculate the average recovery time of shard data;

[0031] If the average recovery time of the shard data is greater than the preset recovery time, it is determined that the transmission validity of the shard data does not meet the requirements, and the transmission rate of the shard data is increased.

[0032] Furthermore, the increase range of the transmission rate of the sliced ​​data is determined by the difference between the average recovery time of the sliced ​​data and the preset recovery time.

[0033] Compared with the prior art, the beneficial effect of the present invention lies in that the method of the present invention adjusts the byte amount in the overlapping area of ​​the shards according to the variance of the proportion of the lost bytes of the shard data. Since part of the data block is lost when the network is interrupted, the shards cannot be correctly aligned with the data blocks, resulting in inaccurate shard boundary processing, which in turn results in the inability to perform proper sharding when sharding the original file, thereby causing partial data loss. By increasing the byte amount in the overlapping area of ​​the shards, more overlapping data can be provided, thereby increasing the accuracy and success rate of alignment. By initially adjusting the transmission rate of the shard data according to the average delay time of the shard data transmission, since a large amount of data may be resumed from breakpoint in a short period of time, resulting in conflicts in part of the data, errors occur in the data transmission process, which in turn causes the loss of breakpoint information, thereby causing unsuccessful data recovery. By reducing the transmission rate of the shard data, the data is reduced. The probability of errors and conflicts occurring during the transmission process is improved, and the accuracy and reliability of data transmission are improved. The life cycle of the shard data is adjusted according to the proportion of the number of successful resumed transfers of the original file. Since the life cycle of some types of files may be longer, the cached data is retained for a long time and is not cleaned up in time, resulting in insufficient storage space, and some resumed data is not stored successfully. By reducing the life cycle of the shard data, the storage space can be released faster, and the probability of resumed transfer failure due to insufficient storage space is reduced. The transmission rate of the shard data is adjusted secondary according to the average recovery time of the shard data. Since the transmission rate of the shard data is too small, the time of data in the resumed transfer process increases, resulting in a decrease in processing efficiency, thereby causing the overall efficiency to decrease when processing large amounts of data. By increasing the transmission rate of the shard data, the speed of data transmission is increased, the time of data recovery is shortened, and the stability of breakpoint resume is improved.

[0034] Furthermore, the method of the present invention adjusts the byte size of the overlapping area of ​​the shards by setting a preset first variance and a preset second variance. Since part of the data of the data block is lost when the network is interrupted, the shards cannot be correctly aligned with the data blocks, resulting in inaccurate shard boundary processing, which in turn leads to the inability to perform proper sharding when sharding the original file, thereby causing partial data loss. By increasing the byte size of the overlapping area of ​​the shards, more overlapping data can be provided, thereby increasing the accuracy and success rate of the alignment, and further achieving improved stability for breakpoint resumption.

[0035] Furthermore, the method of the present invention performs an initial adjustment on the transmission rate of the fragmented data by setting a preset first delay time and a preset second delay time. Since a large amount of data may be resumed in a short period of time, causing conflicts in some data, errors in the data during transmission, and further loss of breakpoint information, resulting in unsuccessful data recovery, by reducing the transmission rate of the fragmented data, the probability of errors and conflicts in the data transmission process is reduced, the accuracy and reliability of data transmission are improved, and the stability of breakpoint resumption is further improved.

[0036] Furthermore, the method of the present invention adjusts the life cycle of the shard data by setting a preset ratio. Since the life cycle of some types of files may be relatively long, the cached data may be retained for a long time without being cleaned up in time, resulting in insufficient storage space and some resumed data not being stored successfully. By reducing the life cycle of the shard data, the storage space can be released faster, reducing the probability of resume failure due to insufficient storage space, and further achieving improved stability for breakpoint resume.

[0037] Furthermore, the method of the present invention performs secondary adjustment on the transmission rate of the sharded data by setting a preset recovery time. Since the transmission rate of the sharded data is too low, the time for data to be resumed increases, resulting in a decrease in processing efficiency, thereby causing a decrease in the overall efficiency when processing large amounts of data. By increasing the transmission rate of the sharded data, the speed of data transmission is increased, the time for data recovery is shortened, and the stability of breakpoint resumption is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is an overall flow chart of a method for implementing breakpoint-resume transmission based on offline synchronization according to an embodiment of the present invention;

[0039] Figure 2 A specific flow chart of the process of adjusting the byte amount of the overlapping area of ​​the fragments in the method for implementing breakpoint resume based on offline synchronization according to an embodiment of the present invention;

[0040] Figure 3 A specific flow chart of the process of initially adjusting the transmission rate of fragmented data in the method for implementing breakpoint resume based on offline synchronization according to an embodiment of the present invention;

[0041] Figure 4 The present invention is a logical flow chart of a method for implementing breakpoint resume based on offline synchronization according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0043] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0044] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0045] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0046] See also Figure 1 , Figure 2 , Figure 3 as well as Figure 4 As shown, they are respectively an overall flow chart of a method for implementing breakpoint-resume transmission based on offline synchronization according to an embodiment of the present invention, a specific flow chart of a process for adjusting the byte amount of a shard overlap area, a specific flow chart of a process for initially adjusting the transmission rate of shard data, and a logic flow chart. A method for implementing breakpoint-resume transmission based on offline synchronization according to the present invention comprises:

[0047] Step S1, fragmenting the original file to output a plurality of fragmented data, and transmitting the plurality of fragmented data offline to a target location when a network is interrupted;

[0048] Step S2, transmitting the fragmented data according to the breakpoints of the fragmented data when the network is interrupted, and merging the fragmented data transmitted to the target location in the order of the fragments to restore the original file;

[0049] Step S3, obtaining the amount of lost bytes of the shard data in several recovery cycles;

[0050] Step S4, determining the stability of breakpoint resume based on the variance of the percentage of lost bytes of the shard data;

[0051] Step S5, if it is determined that the stability does not meet the requirement, the byte amount of the shard overlap area is adjusted or a process for determining the reliability of the shard data retransmission is performed;

[0052] Step S6: If it is determined that the reliability does not meet the requirement, the transmission rate of the shard data is initially adjusted or the life cycle of the shard data is adjusted based on the proportion of the number of successful resumed transmissions of the original file;

[0053] Step S7, obtaining the recovery time of the shard data after a single cycle of transmission according to the transmission rate of the shard data adjusted initially;

[0054] Step S8: secondary adjust the transmission rate of the shard data based on the average recovery time of the shard data.

[0055] Specifically, original files include text files, image files, and audio files.

[0056] Specifically, the fragment data includes the fragment time point, the number of bytes contained in the fragment area, and the fragment location information in the original file.

[0057] Specifically, the location information of the fragment in the original file includes the size of the fragment area, the offset of the fragment position, and the original file path.

[0058] Specifically, the slice overlap area refers to an area where there are overlapped parts between adjacent slice areas.

[0059] Specifically, the successful resumption of the original file transmission means that when an interruption occurs during the transmission process, the remaining data is retransmitted and merged with the transmitted data, and finally the transmission of the entire file is completed.

[0060] In implementation, the method of the present invention adjusts the byte volume of the overlapping area of ​​the shards according to the variance of the proportion of the lost bytes of the shard data. Since part of the data block is lost when the network is interrupted, the shards cannot be correctly aligned with the data blocks, resulting in inaccurate shard boundary processing, which in turn results in the inability to properly shard the original file when sharding, thereby causing partial data loss. By increasing the byte volume of the overlapping area of ​​the shards, more overlapping data can be provided, thereby increasing the accuracy and success rate of alignment. By initially adjusting the transmission rate of the shard data according to the average delay time of the shard data transmission, since a large amount of data may be resumed from breakpoints in a short period of time, conflicts may occur in some data, resulting in data errors during transmission, which in turn results in loss of breakpoint information, resulting in unsuccessful data recovery. By reducing the transmission rate of the shard data, the occurrence of data errors during transmission is reduced. The probability of errors and conflicts is improved, and the accuracy and reliability of data transmission are improved. The life cycle of the shard data is adjusted according to the proportion of the number of successful resumed transfers of the original file. Since the life cycle of some types of files may be long, the cached data is retained for a long time and not cleared in time, resulting in insufficient storage space, and some resumed data is not stored successfully. By reducing the life cycle of the shard data, the storage space can be released faster, and the probability of resumed transfer failure due to insufficient storage space is reduced. The transmission rate of the shard data is adjusted secondary according to the average recovery time of the shard data. Since the transmission rate of the shard data is too small, the time of data in the resumed transfer process increases, resulting in a decrease in processing efficiency, thereby causing the overall efficiency to decrease when processing large amounts of data. By increasing the transmission rate of the shard data, the speed of data transmission is increased, the time of data recovery is shortened, and the stability of breakpoint resume is improved.

[0061] Specifically, the original file resuming transmission method includes fragment resuming transmission, data row-level resuming transmission and message queue data resuming transmission.

[0062] Specifically, row-level data resuming means that during breakpoint resuming, only the data rows whose positions or data volumes have changed are transferred.

[0063] Specifically, the meaning of message queue data resumption is to use the message queue to store and transfer the data to be transmitted during the breakpoint resumption process.

[0064] Specifically, when the resuming mode is row-level resuming, the source table data is queried in the select...where id>{100} manner to achieve the resuming goal; when the resuming mode is message queue data resuming, after storing the offset, the source table data is queried in the select...where id>{100} manner to achieve the resuming goal; when the resuming mode is shard resuming, during recovery, the data stored in the checkpoint is obtained and matched according to the key of the subtask index, that is, subI ndex. If the match is successful, it means that the synchronization has been successful and no further synchronization will be performed. If the match is unsuccessful, it means that the last synchronization was not performed or the synchronization failed and needs to be resynchronized.

[0065] Specifically, the step of adjusting the byte amount of the overlapping area of ​​the slices includes:

[0066] Based on the variance of the percentage of lost bytes of shard data in several recovery cycles, it is determined that the stability of breakpoint-resume transmission does not meet the requirements;

[0067] If the variance of the percentage of lost bytes of the shard data meets the first variance condition or the second variance condition, it is determined that the stability of breakpoint resume transmission does not meet the requirement;

[0068] If the variance of the percentage of lost bytes of the shard data only satisfies the second variance condition, increasing the amount of bytes in the overlapping area of ​​the shards;

[0069] Among them, the first variance condition is that the variance of the proportion of lost bytes of the shard data is greater than the preset first variance and less than or equal to the preset second variance; the second variance condition is that the variance of the proportion of lost bytes of the shard data is greater than the preset second variance.

[0070] It can be understood that the three intervals corresponding to the preset first variance Q1 and the preset second variance Q2 correspond to three situations respectively. The first situation is Q≤Q1, and the stability of the breakpoint resume is judged to meet the requirements; the second situation is Q1<Q≤Q2, and a large amount of data may be resumed in a short period of time, resulting in conflicts in some data, resulting in errors in the data transmission process, and then resulting in the loss of breakpoint information, resulting in unsuccessful data recovery; the third situation is Q>Q2, and part of the data block is lost when the network is interrupted, resulting in the inability to achieve correct alignment of the fragments when aligning with the data blocks, resulting in inaccurate fragment boundary processing, and then resulting in the inability to perform appropriate fragmentation when fragmenting the original file, resulting in partial data loss. In practice, the range of Q1 is generally selected from [0.1, 0.15], and the range of Q2 is generally selected from [0.16, 0.25].

[0071] Preferably, the first variance Q1 is preset to be 0.13, and the second variance Q2 is preset to be 0.2.

[0072] Specifically, the variance of the percentage of lost bytes in the shard data is recorded as Q.

[0073] Specifically, the variance of the percentage of lost bytes of shard data is the variance of the percentage of lost bytes of shard data within several recovery cycles. The calculation method of the variance of the percentage of lost bytes of shard data is a conventional technical means well known to technical personnel in this field. Therefore, the calculation process of the variance of the percentage of lost bytes of shard data will not be repeated here.

[0074] Specifically, the calculation formula for the percentage of lost bytes in shard data is:

[0075]

[0076] Among them, G is the percentage of lost bytes of shard data, K1 is the number of bytes lost in shard data, and K2 is the total number of bytes of shard data.

[0077] In implementation, the method of the present invention determines the stability of breakpoint resume by setting a preset first variance and a preset second variance, thereby reducing the impact of decreased stability of offline synchronization due to inaccurate determination of the stability of breakpoint resume, and further improving the stability of breakpoint resume.

[0078] Specifically, the process of determining the reliability of the resumed transmission of the sharded data includes: when the variance of the proportion of lost bytes of the sharded data only satisfies the first variance condition, preliminarily determining that the reliability of the resumed transmission of the sharded data does not meet the requirements, and performing a secondary determination on the reliability of the resumed transmission of the sharded data based on the average delay time of the sharded data transmission.

[0079] Specifically, the increase in the amount of bytes in the overlapping area of ​​the fragments is determined by the difference between the variance of the proportion of the amount of lost bytes of the fragment data and a preset second variance.

[0080] Specifically, the calculation formula for the number of bytes in the increased shard overlap area is:

[0081]

[0082] Among them, V' is the byte volume of the increased shard overlapping area, V is the byte volume of the current shard overlapping area, A is the minimum unit variance for adjusting the byte volume of the shard overlapping area, and v is the byte volume of the shard overlapping area when the variance of the percentage of lost bytes of shard data is the minimum unit variance.

[0083] In practice, the value of A is generally 0.3 to 0.5, and preferably, the preferred embodiment of A is 0.4.

[0084] In implementation, the optional value range of v is [350Byte, 450Byte]. Preferably, when the byte size of the shard overlap area is less than 400Byte, some data may be lost. Therefore, the preferred embodiment of v is 400Byte.

[0085] Specifically, when the value of V' is a decimal, the value of V' is automatically rounded up to an integer.

[0086] For example, a text document is transmitted using a method based on offline synchronization to achieve breakpoint resuming, where V = 500 Byte, Q = 0.4, and the calculation is:

[0087] In implementation, the method of the present invention adjusts the byte amount of the overlapping area of ​​the shards by setting a preset first variance and a preset second variance. Since part of the data of the data block is lost when the network is interrupted, the shards cannot be correctly aligned with the data blocks, resulting in inaccurate shard boundary processing, which in turn leads to the inability to perform proper sharding when sharding the original file, thereby causing partial data loss. By increasing the byte amount of the overlapping area of ​​the shards, more overlapping data can be provided, thereby increasing the accuracy and success rate of the alignment, and further achieving improved stability for breakpoint resumption.

[0088] Specifically, the step of initially adjusting the transmission rate of the slice data includes:

[0089] Obtain the delay time of shard data transmission in several recovery cycles, and calculate the average delay time of shard data transmission;

[0090] Compare the average delay duration of the segmented data transmission with the preset first delay duration and the preset second delay duration respectively;

[0091] If the average delay time of the segment data transmission is greater than the preset first delay time, it is determined that the reliability of the segment data retransmission does not meet the requirement;

[0092] If the average delay time for transmitting the segmented data is greater than the preset first delay time and less than or equal to the preset second delay time, the transmission rate of the segmented data is reduced;

[0093] If the average delay time of the shard data transmission is greater than the preset second delay time, it is preliminarily determined that the storage validity of the shard data does not meet the requirements, and a secondary determination is made on the storage validity of the shard data based on the proportion of successful resumed transmissions of the original files.

[0094] It can be understood that the three intervals corresponding to the preset first delay time P1 and the preset second delay time P2 correspond to three situations respectively. The first situation is P≤P1, and the reliability of the secondary determination of the fragmented data retransmission meets the requirements; the second situation is P1<P≤P2, and a large amount of data may be retransmitted from a breakpoint in a short period of time, resulting in conflicts in some data, resulting in data errors during transmission, and then resulting in loss of breakpoint information, resulting in unsuccessful data recovery; the third situation is P>P2, and the life cycle of some types of files may be long, resulting in cached data being retained for a long time without being cleared in time, resulting in insufficient storage space, resulting in some retransmitted data not being stored successfully. In practice, the range of P1 is generally [1s, 3s], and the range of P2 is generally [3.5s, 4.5s].

[0095] Preferably, the first delay time length P1 is preset to be 2s, and the second delay time length P2 is preset to be 4s.

[0096] Specifically, the average delay duration of shard data transmission is denoted as P.

[0097] In implementation, the method of the present invention performs a secondary judgment on the reliability of the resumed transmission of fragmented data by setting a preset first delay period and a preset second delay period, thereby reducing the impact of the decreased stability of the resumed transmission due to inaccurate secondary judgment on the reliability of the resumed transmission of fragmented data, and further achieving improved stability of the resumed transmission.

[0098] Specifically, the calculation formula for the average delay time of the slice data transmission is:

[0099]

[0100] Among them, Z is the average delay time of shard data transmission, X a is the delay duration of shard data transmission in the ath recovery cycle, n is the number of recovery cycles, and n is a natural number greater than or equal to 1.

[0101] Specifically, the reduction range of the transmission rate of the slice data is determined by the difference between the average delay duration of the slice data transmission and the preset first delay duration.

[0102] Specifically, the calculation formula for the reduced transmission rate of the fragmented data is:

[0103]

[0104] Among them, H' is the reduced transmission rate of the sliced ​​data, H is the current transmission rate of the sliced ​​data, B is the minimum unit delay duration for adjusting the transmission rate of the sliced ​​data, and h is the transmission rate of the sliced ​​data when the average delay duration of the sliced ​​data transmission is the minimum unit delay duration.

[0105] In practice, the value of B is generally 4.5s to 5.5s. Preferably, the preferred embodiment of B is 5s.

[0106] In implementation, the optional range of the value of h is [300 Byte / s, 500 Byte / s]. Preferably, when the transmission rate of the allocated data is greater than 400 Byte / s, data conflicts may occur. Therefore, the preferred embodiment of h is 400 Byte / s.

[0107] Specifically, when the value of H' is a decimal, the value of H' is automatically rounded up to an integer.

[0108] For example, a text document is transmitted using a method based on offline synchronization to achieve breakpoint-resume transmission, where H = 600 Byte / s, P = 6s, and the calculation is:

[0109] In implementation, the method of the present invention performs an initial adjustment on the transmission rate of the fragmented data by setting a preset first delay time and a preset second delay time. Since a large amount of data may be resumed in a short period of time, conflicts may occur in some data, resulting in data errors during transmission, and further resulting in breakpoint information loss, which leads to unsuccessful data recovery. By reducing the transmission rate of the fragmented data, the probability of errors and conflicts in the data transmission process is reduced, the accuracy and reliability of data transmission are improved, and the stability of breakpoint resumption is further improved.

[0110] Specifically, the steps of adjusting the life cycle of the shard data include:

[0111] Secondarily determine the storage validity of the shard data based on the percentage of the number of successfully resumed original files;

[0112] If the proportion of the number of successfully resumed original files is greater than the preset proportion, it is determined that the storage validity of the shard data does not meet the requirements, and the life cycle of the shard data is reduced.

[0113] It can be understood that the two intervals corresponding to the preset proportion Y0 correspond to two situations. The first situation is Y≤Y0, and the secondary determination of the storage validity of the shard data meets the requirements; the second situation is Y>Y0, and the life cycle of some types of files may be long, resulting in the cache data being retained for a long time without being cleaned up in time, resulting in insufficient storage space, resulting in some continued data not being stored successfully. In practice, Y0 is generally selected in the range of [0.7, 0.9].

[0114] Preferably, the preset proportion Y0=0.8.

[0115] Specifically, the percentage of original files that are successfully resumed is recorded as Y.

[0116] Specifically, the calculation formula for the percentage of original files successfully resumed is:

[0117]

[0118] Among them, S is the ratio of the number of original files successfully resumed, T1 is the number of original files successfully resumed, and T2 is the total number of original files.

[0119] In implementation, the method of the present invention performs a secondary judgment on the storage validity of the sharded data by setting a preset ratio, thereby reducing the impact of the decreased stability of breakpoint-resume transmission due to inaccurate secondary judgment on the storage validity of the sharded data, and further improving the stability of breakpoint-resume transmission.

[0120] Specifically, the reduction extent of the life cycle of the shard data is determined by the difference between the proportion of the number of successfully resumed original files and a preset proportion.

[0121] Specifically, the calculation formula for the reduced life cycle of the shard data is:

[0122]

[0123] Among them, L' is the reduced life cycle of the shard data, L is the current life cycle of the shard data, C is the minimum unit ratio for adjusting the life cycle of the shard data, and l is the life cycle of the shard data when the proportion of the number of successful resumed original files is the minimum unit ratio.

[0124] In practice, the value of C is generally 0.45 to 0.55, and preferably, the preferred embodiment of C is 0.5.

[0125] In implementation, the optional value range of l is [4h, 6h]. Preferably, when the life cycle of the allocated data is greater than 5h, insufficient storage space may be caused. Therefore, the preferred embodiment of l is 5h.

[0126] Specifically, when the value of L' is a decimal, the value of L' is automatically rounded up to an integer.

[0127] For example, a text document is transmitted using a method based on offline synchronization to achieve breakpoint-resume transmission, where L = 6h, Y = 0.9, and the calculation result is

[0128] In implementation, the method of the present invention adjusts the life cycle of shard data by setting a preset ratio. Since the life cycle of some types of files may be relatively long, cached data may be retained for a long time without being cleaned up in time, resulting in insufficient storage space and some resumed data not being stored successfully. By reducing the life cycle of shard data, storage space can be released faster, reducing the probability of resume failure due to insufficient storage space, and further achieving improved stability for breakpoint resume.

[0129] Specifically, the step of secondarily adjusting the transmission rate of the slice data includes:

[0130] Obtain the recovery time of shard data in several recovery cycles, and calculate the average recovery time of shard data;

[0131] If the average recovery time of the shard data is greater than the preset recovery time, it is determined that the transmission validity of the shard data does not meet the requirements, and the transmission rate of the shard data is increased.

[0132] It can be understood that the two intervals corresponding to the preset recovery time R0 correspond to two situations. The first situation is R≤R0, which determines that the transmission validity of the shard data meets the requirements; the second situation is R>R0. Since the transmission rate of the shard data is too low, the time of data retransmission increases, resulting in a decrease in processing efficiency, which causes a decrease in overall efficiency when processing large amounts of data. In practice, R0 is generally selected in the range of [8s, 12s].

[0133] Preferably, the preset recovery time length R0=10s.

[0134] Specifically, the average recovery time of shard data is denoted as R.

[0135] Specifically, the calculation formula for the average recovery time of shard data is:

[0136]

[0137] Among them, R is the average recovery time of shard data, E f is the recovery time of the shard data in the fth recovery cycle, i is the number of recovery cycles, and i is a natural number greater than or equal to 1.

[0138] In implementation, the method of the present invention determines the transmission validity of the fragmented data by setting a preset recovery time, thereby reducing the impact of the stability of breakpoint-resume transmission caused by inaccurate judgment of the transmission validity of the fragmented data, and further improving the stability of breakpoint-resume transmission.

[0139] Specifically, the increase range of the transmission rate of the slice data is determined by the difference between the average recovery time of the slice data and the preset recovery time.

[0140] Specifically, the calculation formula for the increased transmission rate of the fragmented data is:

[0141]

[0142] Wherein, H" is the increased transmission rate of the shard data, D is the minimum unit recovery time for adjusting the transmission rate of the shard data, and h' is the transmission rate of the shard data when the average recovery time of the shard data is the minimum unit recovery time.

[0143] In practice, the value of D is generally 13s to 17s, and preferably, the preferred embodiment of D is 15s.

[0144] In implementation, the optional range of h' is [200 Byte / s, 300 Byte / s]. Preferably, when the transmission rate of the sliced ​​data is less than 250 Byte / s, the transmission efficiency may be reduced. Therefore, the preferred embodiment of h' is 250 Byte / s.

[0145] Specifically, when the value of H" is a decimal, the value of H" is automatically rounded up to an integer.

[0146] For example, a text document is transmitted using a method based on offline synchronization to achieve breakpoint-resume transmission, where H' = 280 Byte / s, R = 14 s, and the calculation result is

[0147] In implementation, the method of the present invention performs secondary adjustment on the transmission rate of the sharded data by setting a preset recovery time. Since the transmission rate of the sharded data is too low, the time for data to be resumed increases, resulting in a decrease in processing efficiency, thereby causing a decrease in the overall efficiency when processing large amounts of data. By increasing the transmission rate of the sharded data, the speed of data transmission is increased, the time for data recovery is shortened, and the stability of breakpoint resumption is further improved.

[0148] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A method for implementing breakpoint-resume transmission based on offline synchronization, characterized in that: The following steps are involved: Slicing the original file to output a plurality of slice data, and transmitting the plurality of slice data offline to a target location when a network is interrupted; The fragmented data is transmitted according to the breakpoint of the fragmented data when the network is interrupted, and the fragmented data transmitted to the target location is merged in the order of the fragments to restore the original file; Get the amount of lost bytes of shard data within several recovery cycles; Determining the stability of breakpoint-resume transmission based on the variance of the percentage of lost bytes of the shard data; If it is determined that the stability does not meet the requirement, the byte amount of the shard overlap area is adjusted or a process for determining the reliability of the continued transmission of the shard data is performed; If it is determined that the reliability does not meet the requirement, the transmission rate of the shard data is initially adjusted or the life cycle of the shard data is adjusted based on the proportion of the number of successful resumed transmissions of the original file; The recovery time of the shard data is obtained after a single cycle of transmission of the shard data according to the transmission rate of the shard data adjusted initially; Secondary adjustment of the transmission rate of shard data based on the average recovery time of shard data; The step of adjusting the byte amount of the overlapping area of ​​the slices comprises: Based on the variance of the percentage of lost bytes of shard data in several recovery cycles, it is determined that the stability of breakpoint-resume transmission does not meet the requirements; If the variance of the percentage of lost bytes of the shard data meets the first variance condition or the second variance condition, it is determined that the stability of breakpoint resume transmission does not meet the requirement; If the variance of the percentage of lost bytes of the shard data only satisfies the second variance condition, increasing the amount of bytes in the overlapping area of ​​the shards; The first variance condition is that the variance of the percentage of lost bytes of the shard data is greater than the preset first variance and less than or equal to the preset second variance; the second variance condition is that the variance of the percentage of lost bytes of the shard data is greater than the preset second variance; The process of determining the reliability of the fragmented data retransmission includes: when the variance of the percentage of lost bytes of the fragmented data only satisfies the first variance condition, preliminarily determining that the reliability of the fragmented data retransmission does not meet the requirement, and performing a secondary determination on the reliability of the fragmented data retransmission according to the average delay time of the fragmented data transmission; The step of initially adjusting the transmission rate of the sliced ​​data includes: Obtain the delay time of shard data transmission in several recovery cycles, and calculate the average delay time of shard data transmission; Compare the average delay duration of the segmented data transmission with the preset first delay duration and the preset second delay duration respectively; If the average delay time of the segment data transmission is greater than the preset first delay time, it is determined that the reliability of the segment data retransmission does not meet the requirement; If the average delay time for transmitting the segmented data is greater than the preset first delay time and less than or equal to the preset second delay time, the transmission rate of the segmented data is reduced; If the average delay time of the shard data transmission is greater than the preset second delay time, it is preliminarily determined that the storage validity of the shard data does not meet the requirements, and a secondary determination is made on the storage validity of the shard data based on the proportion of the number of successful resumed transmissions of the original files; The reduction range of the transmission rate of the sliced ​​data is determined by the difference between the average delay time of the sliced ​​data transmission and the preset first delay time; The steps of adjusting the life cycle of the shard data include: Secondarily determine the storage validity of the shard data based on the percentage of the number of successfully resumed original files; If the proportion of the number of successfully resumed original files is greater than the preset proportion, it is determined that the storage validity of the shard data does not meet the requirements, and the life cycle of the shard data is reduced.

2. The method for implementing breakpoint-resume transmission based on offline synchronization according to claim 1, characterized in that: The original file resuming transmission method includes fragment resuming transmission, data row level resuming transmission and message queue data resuming transmission.

3. The method for implementing breakpoint-resume transmission based on offline synchronization according to claim 2, characterized in that: The increase in the amount of bytes in the overlapping area of ​​the fragments is determined by the difference between the variance of the proportion of the amount of lost bytes of the fragment data and a preset second variance.

4. The method for implementing breakpoint-resume transmission based on offline synchronization according to claim 3, characterized in that: The calculation formula for the average delay time of the fragment data transmission is: Among them, Z is the average delay time of shard data transmission, X a is the delay duration of shard data transmission in the ath recovery cycle, n is the number of recovery cycles, and n is a natural number greater than or equal to 1.

5. The method for implementing breakpoint-resume transmission based on offline synchronization according to claim 4, characterized in that: The reduction extent of the life cycle of the shard data is determined by the difference between the proportion of the number of successfully resumed original files and a preset proportion.

6. The method for implementing breakpoint-resume transmission based on offline synchronization according to claim 5, characterized in that: The step of secondarily adjusting the transmission rate of the sliced ​​data comprises: Obtain the recovery time of shard data in several recovery cycles, and calculate the average recovery time of shard data; If the average recovery time of the shard data is greater than the preset recovery time, it is determined that the transmission validity of the shard data does not meet the requirements, and the transmission rate of the shard data is increased.

7. The method for implementing breakpoint-resume transmission based on offline synchronization according to claim 6, characterized in that: The increase range of the transmission rate of the slice data is determined by the difference between the average recovery time of the slice data and the preset recovery time.

Images