Data transmission method and device, computer equipment, storage medium and computer program product

Abstract

The invention relates to a data transmission method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: transmitting each first data fragment of target data in parallel; under the condition that the transmission pause event is triggered, target first data fragments which do not complete transmission in the first data fragments and untransmitted data corresponding to the target first data fragments are determined; under the condition of triggering a restart transmission event, re-fragmenting each piece of untransmitted data to obtain each second data fragment, and respectively determining fragment identification information corresponding to each second data fragment; wherein the fragment identifier information at least comprises a target first data fragment identifier, and the target first data fragment identifier is an identifier of a target first data fragment to which data in the second data fragment belongs; and transmitting the second data fragments and the fragment identification information of the second data fragments in parallel. By adopting the method, the data transmission efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data transmission method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the rapid development of software applications, the demand for software artifact management is increasing day by day, and the artifact system has become an indispensable part of the software development process. Since the artifact files are generally large, users usually have the need to pause the transmission at any time when the artifact files are uploaded to complete other tasks.

[0003] Related technologies usually transmit the fragments of the finished file in parallel through multiple threads, and determine which fragment has not been uploaded when the user pauses the transmission, and delete the cache file of the fragment at the data receiving end. When the user restarts the transmission, the data sending end retransmits the entire fragment that has not been uploaded. However, since the data volume of each fragment of the finished file is large, this method of retransmitting the entire fragment has the problem of a large amount of data to be retransmitted and low data transmission efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a data transmission method, apparatus, computer equipment, storage medium and computer program product to address the above technical issues.

[0005] In a first aspect, the present application provides a data transmission method. The method comprises:

[0006] Transmitting each first data slice of the target data to the data receiving end in parallel through multiple threads;

[0007] In the case of triggering a transmission suspension event, terminating the transmission of each of the first data slices, determining the target first data slices that have not completed transmission in each of the first data slices, and the untransmitted data corresponding to each of the target first data slices;

[0008] In the case of triggering a restart transmission event, re-slicing each of the untransmitted data to obtain each second data slice, and respectively determining the slice identification information corresponding to each of the second data slices; wherein the slice identification information at least includes a target first data slice identification, and the target first data slice identification is an identification of the target first data slice to which the data in the second data slice belongs;

[0009] The second data slices and the slice identification information of the second data slices are transmitted to the data receiving end in parallel through multiple threads.

[0010] In one embodiment, the determining of the target first data slices that have not been transmitted in each of the first data slices and the untransmitted data corresponding to each of the target first data slices includes:

[0011] Sending thread end notification information to the data receiving end;

[0012] receiving target data information sent by the data receiving end in response to the thread termination notification information, the target data information including identifiers of each target first data slice that has not been completely transmitted, and identifiers of transmitted data corresponding to each target first data slice;

[0013] Each of the target first data slices is obtained from the target data information, and the untransmitted data corresponding to each of the target first data slices is determined respectively according to the identifier of the transmitted data corresponding to each of the target first data slices.

[0014] In one embodiment, the re-slicing each of the untransmitted data to obtain each second data slice includes:

[0015] Obtaining the number of available threads and network status information, and determining the number of sending threads according to the size of each of the untransmitted data, the number of available threads and the network status information;

[0016] According to the number of sending threads, each of the untransmitted data is re-sliced ​​to obtain each second data slice.

[0017] In one of the embodiments, the network status information includes at least a packet loss rate and a network bandwidth, the number of sending threads is less than or equal to the number of available threads, and the number of sending threads is positively correlated with the size of each untransmitted data and the network bandwidth, and the number of sending threads is negatively correlated with the packet loss rate.

[0018] In one embodiment, re-slicing each of the untransmitted data according to the number of sending threads to obtain each second data slice includes:

[0019] Sequentially concatenate the untransmitted data to obtain target untransmitted data, and equally divide the target untransmitted data into second data slices that are the number of the sending threads.

[0020] In one embodiment, the respectively determining the shard identification information corresponding to each of the second data shards includes:

[0021] Determine each sub-slice contained in each second data slice, and determine the first byte offset of each sub-slice corresponding to the target data and the length of each sub-slice, respectively, and the data in the same sub-slice comes from the untransmitted data corresponding to the same target first data slice;

[0022] For any of the second data slices, generate slice identification information corresponding to the second data slice, the slice identification information at least including the first byte offset corresponding to each of the sub-slices, the length of each of the sub-slices, and the identification of the target first data slice corresponding to each of the sub-slices.

[0023] In one embodiment, the step of respectively determining each sub-shard contained in each second data shard includes:

[0024] respectively determining a second byte offset of each of the untransmitted data in the target untransmitted data, and respectively determining a third byte offset of each of the second data fragments in the target untransmitted data;

[0025] According to the second byte offset and the third byte offset, each sub-shard contained in each second data shard is determined respectively.

[0026] In one embodiment, the method further comprises:

[0027] Displaying transmission progress information in a display interface, wherein the transmission progress information is used to indicate the size of data that has been sent to the data receiving end;

[0028] When receiving the transmission completion information sent by the data receiving end, updating the transmission progress information according to the transmission completion information; the transmission completion information is sent by the data receiving end when the transmission of any of the first data fragments is completed;

[0029] When the transmission suspension event is triggered, the transmission progress information is updated according to the untransmitted data corresponding to each of the target first data segments.

[0030] In a second aspect, the present application also provides a data transmission device. The device comprises:

[0031] A first transmission module, used for transmitting each first data slice of the target data to the data receiving end in parallel through multiple threads;

[0032] A determination module, configured to terminate the transmission of each of the first data slices when a transmission suspension event is triggered, and determine the target first data slices that have not been transmitted in each of the first data slices, and the untransmitted data corresponding to each of the target first data slices;

[0033] A sharding module, configured to, when a transmission restart event is triggered, re-shard each of the untransmitted data to obtain each second data shard, and respectively determine shard identification information corresponding to each of the second data shards; wherein the shard identification information at least includes a target first data shard identification, and the target first data shard identification is an identification of the target first data shard to which the data in the second data shard belongs;

[0034] The second transmission module is used to transmit the second data slices and the slice identification information of the second data slices to the data receiving end in parallel through multiple threads.

[0035] In one embodiment, the determining module is further used to:

[0036] Sending thread end notification information to the data receiving end;

[0037] receiving target data information sent by the data receiving end in response to the thread termination notification information, the target data information including identifiers of each target first data slice that has not been completely transmitted, and identifiers of transmitted data corresponding to each target first data slice;

[0038] Each of the target first data slices is obtained from the target data information, and the untransmitted data corresponding to each of the target first data slices is determined respectively according to the identifier of the transmitted data corresponding to each of the target first data slices.

[0039] In one embodiment, the slicing module is further used for:

[0040] Obtaining the number of available threads and network status information, and determining the number of sending threads according to the size of each of the untransmitted data, the number of available threads and the network status information;

[0041] According to the number of sending threads, each of the untransmitted data is re-sliced ​​to obtain each second data slice.

[0042] In one of the embodiments, the network status information includes at least a packet loss rate and a network bandwidth, the number of sending threads is less than or equal to the number of available threads, and the number of sending threads is positively correlated with the size of each untransmitted data and the network bandwidth, and the number of sending threads is negatively correlated with the packet loss rate.

[0043] In one embodiment, the slicing module is further used for:

[0044] Sequentially concatenate the untransmitted data to obtain target untransmitted data, and equally divide the target untransmitted data into second data slices that are the number of the sending threads.

[0045] In one embodiment, the slicing module is further used for:

[0046] Determine each sub-slice contained in each second data slice, and determine the first byte offset of each sub-slice corresponding to the target data and the length of each sub-slice, respectively, and the data in the same sub-slice comes from the untransmitted data corresponding to the same target first data slice;

[0047] For any of the second data slices, generate slice identification information corresponding to the second data slice, the slice identification information at least including the first byte offset corresponding to each of the sub-slices, the length of each of the sub-slices, and the identification of the target first data slice corresponding to each of the sub-slices.

[0048] In one embodiment, the slicing module is further used for:

[0049] respectively determining a second byte offset of each of the untransmitted data in the target untransmitted data, and respectively determining a third byte offset of each of the second data fragments in the target untransmitted data;

[0050] According to the second byte offset and the third byte offset, each sub-shard contained in each second data shard is determined respectively.

[0051] In one embodiment, the device further comprises:

[0052] A display module, used to display transmission progress information in a display interface, wherein the transmission progress information is used to indicate the size of data sent to the data receiving end;

[0053] A first updating module is configured to update the transmission progress information according to the transmission completion information when receiving the transmission completion information sent by the data receiving end; the transmission completion information is sent by the data receiving end when the transmission of any of the first data fragments is completed;

[0054] The second updating module is used to update the transmission progress information according to the untransmitted data corresponding to each of the target first data slices when the transmission suspension event is triggered.

[0055] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, any one of the above methods is implemented.

[0056] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, any of the above methods is implemented.

[0057] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0058] The above-mentioned data transmission method, apparatus, computer equipment, storage medium and computer program product obtain each target first data slice that has not been transmitted after suspending data transmission, and determine the untransmitted data from each target first data slice. After restarting data transmission, the data sending end further splits the untransmitted data into second data slices again, and generates slice identification information for indicating which target first data slices the data in each second data slice originally belonged to, and transmits the second data slice and the slice identification information together to the data receiving end, so that the data receiving end can refer to the slice identification information and correctly splice the data in the second data slice with the data received in the original target first data slice. Therefore, the amount of data that needs to be retransmitted each time the transmission is restarted can be reduced, and at the same time, it can ensure that the retransmitted data and the transmitted data are correctly spliced, thereby improving the data transmission efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A schematic diagram of a flow chart of a data transmission method in an embodiment;

[0060] Figure 2 is a flow chart of step 104 in one embodiment;

[0061] Figure 3 is a flow chart of step 106 in one embodiment;

[0062] Figure 4 is a flow chart of step 106 in one embodiment;

[0063] Figure 5 is a flow chart of step 402 in one embodiment;

[0064] Figure 6 A schematic diagram of a process for displaying transmission progress information in an embodiment;

[0065] Figure 7 A schematic diagram of a flow chart of a data transmission method in an embodiment;

[0066] Figure 8 is a structural block diagram of a data transmission device in one embodiment;

[0067] Fig. 9FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0069] In one embodiment, Figure 1 As shown, a data transmission method is provided. This embodiment takes the method applied to a data sending end as an example, and includes the following steps:

[0070] Step 102: Transmitting each first data fragment of the target data to the data receiving end in parallel through multiple threads.

[0071] In an embodiment of the present application, the target data is a file that needs to be uploaded to the back end (data receiving end). When sending the target data, the data sending end first splits the target data into multiple first data slices. The present application does not specifically limit the method for splitting the target data. For example, the data slice size can be pre-set, and the target data can be split into multiple first data slices according to the data slice size, so that the size of each first data slice is the pre-set data slice size; or the number of slices can be pre-set, and the target data can be split into a pre-set number of first data slices, etc.

[0072] After obtaining each first data slice, the data sending end transmits each first data slice to the data receiving end in parallel through multiple threads. When the number of first data slices is not more than the number of available threads, the data sending end starts to transmit all first data slices at the same time. When the number of first data slices is more than the number of available threads, the data sending end transmits each first data slice to the data receiving end in multiple rounds, and transmits a part of each first data slice in each round of transmission.

[0073] Step 104, when the transmission suspension event is triggered, terminate the transmission of each first target data slice, determine the target first data slice that has not been transmitted in each first data slice, and the untransmitted data corresponding to each target first data slice.

[0074] In the embodiment of the present application, the transmission suspension event may be an event detected by the data sending end, such as a user triggering a transmission suspension control on the display interface of the data sending end. The transmission suspension event may also be an event detected by the data receiving end. For example, when the data receiving end determines that the current network quality is poor, resulting in poor transmission quality of each first data segment (for example, a high packet loss rate, slow transmission speed, etc.), it may determine to trigger the transmission suspension event and notify the data sending end of the occurrence of the transmission suspension event.

[0075] When a transmission suspension event is triggered, the data sending end stops transmitting each first data slice through each thread, and determines the target first data slices that have not been transmitted, and the untransmitted data in each target first data slice that has not been transmitted to the data receiving end.

[0076] When transmitting the first data segment, the data sending end can record in real time which part of the first data segment is currently being transmitted. For example, when sending a data packet of the first data segment, the data sending end can determine the byte offset of the data contained in the data packet in the first data segment. When the transmission suspension event is triggered, the data sending end can obtain each first data segment currently being transmitted, and determine whether the first data segment has completed transmission based on the total number of bytes of the first data segment and the byte offset recorded in the last data packet sent before the transmission suspension event is triggered, and if not, which part has not completed transmission.

[0077] If the data transmitting end transmits the first data slice to the data receiving end in multiple rounds, the data transmitting end, in addition to determining the target first data slice that has not been transmitted from the first data slice that is currently being transmitted through the above method, also needs to use each first data slice that has not started to be transmitted as the target first data slice. The untransmitted data corresponding to each first data slice that has not started to be transmitted is all the data in these first data slices.

[0078] Step 106, when a transmission restart event is triggered, each untransmitted data is re-segmented to obtain each second data segment, and the segment identification information corresponding to each second data segment is determined respectively; wherein the segment identification information at least includes the target first data segment identification, and the target first data segment identification is the identification of the target first data segment to which the data in the second data segment belongs.

[0079] In the embodiment of the present application, the restart transmission event is an event relative to the pause transmission event. For example, in the case where the pause transmission event is a user triggering the pause transmission control on the display interface of the data sending end, the restart transmission event is a user triggering the restart transmission control on the display interface of the data sending end. In the case where the pause transmission event is a poor network quality, the restart transmission event may be that the data receiving end detects that the network quality has improved, for example, when the data receiving end communicates with other data sending ends, the data receiving end detects that the packet loss rate or data transmission speed meets the requirements.

[0080] After the restart transmission event is triggered, the data sending end re-slices the untransmitted data corresponding to each target first data slice. The number of slices during re-slicing can be determined according to the current network status, the current number of available threads, etc.

[0081] Re-slicing may be to slice each untransmitted data separately, that is, when the target first data slices are slices A and B, the slices obtained after re-slicing may be slices A1, A2, B1, B2, wherein the data in A1 and A2 are all from A, and the data in B1 and B2 are all from B. Re-slicing may also be to slice each untransmitted data together after mixing them, that is, when the target first data slices are slices A and B, the slices obtained after re-slicing may be slices C1 and C2, wherein the data in C1 may be from both A and B, and the data in C2 may also be from both A and B.

[0082] After re-slicing to obtain the second data slices, it is necessary to generate slice identification information for each second data slice. The slice identification information at least includes the identification of each target first data slice, which is used to indicate which part of the data in each second data slice comes from which target first data slice. This allows the data receiving end to correctly combine the data in the second data slice with the transmitted data in the target first data slice previously received by the data receiving end after reading the slice identification information, so as to obtain the complete target data.

[0083] It should be noted that if the re-sharding strategy adopted will fragment the untransmitted data from the same target first data fragment into different second data fragments, the fragment identification information may also include a splicing order identifier for each target first data fragment identifier, the splicing order identifier is used to indicate how many second data fragments the untransmitted data in the target first data fragment is specifically divided into, and when the data corresponding to the target first data fragment in each second data fragment is spliced ​​to obtain the untransmitted data of the target first data fragment, the data in the second data fragment corresponding to the target first data fragment corresponds to the splicing order in the splicing.

[0084] For example, if the untransmitted data of the target first data slice A is sequentially split into A1, A2, and A3, and A1, A2, and A3 belong to different second data slices, then for the second data slice where A1 is located, the slice identification information corresponding to the second data slice should record the identifier of the target first data slice A (assuming that A is used as the identifier), and the splicing order identifier for identifier A (recording that identifier A corresponds to 3 second data slices, and the splicing order corresponding to this second data slice is 1). In this way, after the data receiving end reads the slice identification information of each second data slice, if it is necessary to obtain the untransmitted data of the target first data slice A, the data receiving end can obtain the second data slices containing identifier A in each slice identification information, and sequentially splice the data corresponding to identifier A in these second data slices according to the splicing order identifier in these second data slices to obtain the untransmitted data of the target first data slice A.

[0085] Step 108: Transmit each second data slice and the slice identification information of each second data slice to the data receiving end in parallel through multiple threads.

[0086] In the embodiment of the present application, the data transmitting end transmits each second data slice and the slice identification information of each second data slice to the data receiving end in parallel. The data transmitting end may preferentially send the slice identification information of each second data slice to the data receiving end before sending the second data slice, so that the data receiving end can refer to the slice identification information in real time during the process of receiving each second data slice, and combine the received data with the transmitted data cached in the data receiving end.

[0087] It should be noted that when the data sending end sends each first data fragment before, it can simultaneously transmit the characteristic information of each first data fragment to the data receiving end. The characteristic information is used to verify the integrity of the first data fragment. For example, the characteristic information can be the md5 (Message-Digest Algorithm 5, a hash function) value or any digital signature of the first data fragment. After determining that the transmission of the first data fragment is completed, the data receiving end can calculate the characteristic information of the received first data fragment, and compare the calculated characteristic information with the characteristic information sent by the data sending end to determine whether the data receiving end has received the complete first data fragment.

[0088] After parsing each second data slice at the data receiving end and combining the data in each second data slice with the transmitted data of the target first data slice to which it originally belongs, the data receiving end may also calculate the characteristic information of the combined data, and compare the calculated characteristic information with the characteristic information sent by the data sending end for the target first data slice to determine whether the target first data slice received by the data receiving end is complete. If it is incomplete, the data receiving end may notify the data sending end to resend the target first data slice, or may notify the data sending end to resend each second data slice carrying the untransmitted data of the target first data slice.

[0089] The data transmission method provided in the embodiment of the present application obtains each target first data slice that has not been transmitted after suspending data transmission, and determines the untransmitted data from each target first data slice. After restarting data transmission, the data sending end further splits the untransmitted data into second data slices again, and generates slice identification information for indicating which target first data slices the data in each second data slice originally belonged to, and transmits the second data slice and the slice identification information together to the data receiving end, so that the data receiving end can refer to the slice identification information and correctly splice the data in the second data slice with the data received in the original target first data slice. Therefore, the amount of data that needs to be retransmitted each time the transmission is restarted can be reduced, and at the same time, it can ensure that the retransmitted data and the transmitted data are correctly spliced, thereby improving the efficiency of data transmission.

[0090] In one embodiment, Figure 2 As shown, in step 104, determining the target first data slices that have not completed transmission in each first data slice, and the untransmitted data corresponding to each target first data slice, includes:

[0091] Step 202: Send thread termination notification information to the data receiving end.

[0092] Step 204: Receive the target data information sent by the data receiving end in response to the thread termination notification information, wherein the target data information includes the identifiers of each target first data slice that has not been completely transmitted, and the identifiers of the transmitted data corresponding to each target first data slice.

[0093] Step 206: Acquire each target first data slice from the target data information, and determine the untransmitted data corresponding to each target first data slice according to the identifier of the transmitted data corresponding to each target first data slice.

[0094] In the embodiment of the present application, considering that in the case of network fluctuations and other problems, the data sent by the data sending end may not arrive at the data receiving end, so the data sending end may not accurately determine which data is not transmitted data. The data sending end may instruct the data receiving end to determine the target first data slice and which data in the target first data slice is not transmitted data.

[0095] When the transmission suspension event is triggered, the data transmitting end sends thread termination notification information to the data receiving end to inform the data receiving end that the data transmitting end has terminated each thread for transmitting data. After receiving the thread termination notification information, the data receiving end determines which first data fragments have completed transmission.

[0096] The embodiment of the present application does not specifically limit how the data receiving end determines which first data fragments have completed transmission. The specific determination method can be set according to the data transmission format agreed upon between the data sending end and the data receiving end. For example, in each data packet sent by the data sending end to the data receiving end, the total number of data packets corresponding to the first data fragments and the number of the currently transmitted data packet in the total number are carried. The data receiving end can determine whether the first data fragment has been transmitted based on whether the number of the last received data packet before receiving the thread end notification information is equal to the total number of data packets. Or in the case where the data sending end has sent the characteristic information of the first data fragment to the data receiving end, the data receiving end can calculate the characteristic information of each data currently received for the first data fragment, and determine whether the first data fragment has been transmitted based on whether the characteristic information matches the characteristic information sent by the data sending end.

[0097] After the data receiving end determines the target first data segments that have not been transmitted, it obtains the identifiers of these target first data segments. The data receiving end also needs to determine which data in these target first data segments are identifiers that can be used to characterize which data are transmitted data. For example, the data receiving end can calculate the number of bytes of the received data and use the number of bytes as the identifier of the transmitted data, or when the data packet carries a number, the data receiving end can also use the number of each received data packet as the identifier of the transmitted data.

[0098] The data receiving end then sends the identifier of the target first data segment and the identifier of the transmitted data to the data sending end as target data information.

[0099] After receiving the target data information, the data transmitting end determines the target first data slice according to the identifier of the target first data slice. The data transmitting end can also determine the untransmitted data according to the identifier of the transmitted data. For example, when the identifier of the transmitted data is the number of bytes, the data transmitting end can take the data of the first data slice starting from the starting position and the length of the number of bytes as the transmitted data, and the data other than the transmitted data is the untransmitted data. When the identifier of the transmitted data is the number of each received data packet, the data transmitting end can determine the number of each untransmitted data packet according to the information, and take the data in these data packets as the untransmitted data.

[0100] The data transmission method provided by the embodiment of the present application enables the data receiving end to determine the target first data fragments that have not completed transmission, and determines the transmitted data in each target first data fragment, so that the data sending end can determine the untransmitted data based on the transmitted data. Determining the untransmitted data based on the data actually received by the data receiving end can reduce the data sent by the data sending end due to network fluctuations, which is different from the data received by the data receiving end, causing the data sending end to misjudge the untransmitted data, resulting in the data receiving end failing to verify the integrity of the target data, and requiring the data sending end to retransmit the data. The efficiency of data transmission can be further improved.

[0101] In one embodiment, Figure 3 As shown, in step 106, each untransmitted data is re-sliced ​​to obtain each second data slice, including:

[0102] Step 302, obtaining the number of available threads and network status information, and determining the number of sending threads according to the size of each untransmitted data, the number of available threads and the network status information.

[0103] Step 304: Re-segment each untransmitted data according to the number of sending threads to obtain each second data segment.

[0104] In an embodiment of the present application, when re-sharding, the data sending end can determine the number of sending threads based on the number of currently available threads, the size of the untransmitted data and the network status, and obtain each second data fragment according to the number of sending threads, so that the data sending end can transmit all the second data fragments in parallel at the same time through the sending number of threads, thereby improving the efficiency of data transmission.

[0105] The total size of the untransmitted data can be calculated based on the sum of the sizes of the untransmitted data. The number of sending threads can be positively correlated with the total size, so that when the untransmitted data is large, the data sending end can use more threads to simultaneously send each second data fragment, thereby improving data transmission efficiency.

[0106] The network status may include at least a packet loss rate and a network bandwidth. The packet loss rate may be negatively correlated with the number of sending threads. When the packet loss rate is high, the probability that the data sending end needs to process a retransmission request sent by the data receiving end when the packet loss is found is also high. Therefore, the data sending end may determine a smaller number of sending threads so that the data sending end can allocate a part of the resources to process the retransmission request, thereby avoiding overloading of the data sending end.

[0107] The network bandwidth may be positively correlated with the number of sending threads. When the bandwidth is low, the data sender may also determine a smaller number of sending threads to avoid the network status being deteriorated due to excessive amount of data being transmitted at the same time.

[0108] The number of available threads is determined based on the current load of the data sender. The data sender can determine how many resources it can allocate to send untransmitted data based on the current load status, and determine the maximum number of threads for sending based on the resources that can be allocated, that is, the number of available threads. The number of sending threads should be less than or equal to the number of available threads.

[0109] The specific number of available threads can be determined by those skilled in the art according to the above principles and actual needs.

[0110] The data transmission method provided in the embodiment of the present application determines the number of sending threads according to the number of available threads and network status information, and performs data sharding according to the number of sending threads. Each shard can be sent simultaneously using a number of threads suitable for the current network status, thereby improving data transmission efficiency.

[0111] In one embodiment, each untransmitted data is re-sliced ​​according to the number of sending threads to obtain each second data slice, including:

[0112] The untransmitted data are sequentially spliced ​​to obtain target untransmitted data, and the target untransmitted data is equally divided into second data slices that are the number of sending threads.

[0113] In the embodiment of the present application, after the untransmitted data is sequentially spliced, the target untransmitted data can be obtained. The target untransmitted data is equally divided according to the number of sending threads to obtain each second data slice. Through such a method, it can be ensured that the size of each second data slice is roughly the same, so that when each second data slice is sent in parallel, each second data slice can be transmitted in roughly the same time. At the same time, the untransmitted data of each target first data slice can be dispersed as much as possible into fewer second data slices, so that the data receiving end can extract the untransmitted data from the second data slice.

[0114] In one embodiment, Figure 4 As shown, in step 106, the shard identification information corresponding to each second data shard is determined respectively, including:

[0115] Step 402, determining each sub-slice contained in each second data slice, and determining the first byte offset of the target data corresponding to each sub-slice and the length of each sub-slice, wherein the data in the same sub-slice comes from the untransmitted data corresponding to the same target first data slice;

[0116] Step 404: for any second data slice, generate slice identification information corresponding to the second data slice, the slice identification information at least including the first byte offset corresponding to each sub-slice, the length of each sub-slice, and the identification of the target first data slice corresponding to each sub-slice.

[0117] In the embodiment of the present application, when determining the slice identification information corresponding to the second data slice, the sub-slices contained in the second data slice can be first determined. The sub-slice refers to the portion of the second data slice that corresponds to the same target first data slice. Since each second data slice is obtained by sequentially splicing and segmenting the untransmitted data of each target first data slice, the portion corresponding to the same target first data slice in the second data slice will be a portion of continuous data.

[0118] When determining the untransmitted data of each target first data slice, the data receiving end may simultaneously determine the byte offset of the untransmitted data relative to the target data, where the byte offset is used to indicate the byte number from the beginning of the target data where the beginning of the untransmitted data is located. When segmenting the second data slice, the data receiving end may further determine the first byte offset of each sub-slice in the second data slice relative to the target data based on which byte from the beginning of the untransmitted data the segmentation point is located.

[0119] According to the first byte offset corresponding to each sub-slice and the length of each sub-slice, the slice identification information can be generated. The slice identification information includes the identification of the target first data slice corresponding to each sub-slice, the length of each sub-slice, and the first byte offset of each sub-slice for the target data, so that the data receiving end can read the complete sub-slice from the second data slice according to the length of the sub-slice, determine which transmitted data needs to be spliced ​​with the sub-slice according to the identification of the target first data slice corresponding to each sub-slice, and can also determine the order of splicing each sub-slice according to the first byte offset of each sub-slice.

[0120] In one embodiment, Figure 5 As shown, in step 402, each sub-shard contained in each second data shard is determined respectively, including:

[0121] Step 502, respectively determining the second byte offset of each untransmitted data in the target untransmitted data, and respectively determining the third byte offset of each second data fragment in the target untransmitted data;

[0122] Step 504: Determine each sub-shard contained in each second data shard according to the second byte offset and the third byte offset.

[0123] In the embodiment of the present application, the second byte offset of the untransmitted data in the target untransmitted data is used to indicate the byte number at which the beginning of the untransmitted data is located from the beginning of the target untransmitted data. The third byte offset of the second data segment in the target untransmitted data is used to indicate the byte number at which the beginning of the second data segment is located from the beginning of the target untransmitted data.

[0124] For a certain second data slice, according to the third byte offset of the second data slice and the third byte offset of the second data slice next to the second data slice, the byte range covered by the second data slice in the target untransmitted data can be determined. Similarly, the byte range covered by each untransmitted data can also be determined. For a certain second data slice, according to the byte range covered by the untransmitted data that overlaps with the byte range covered by the second data slice, it can be determined which untransmitted data the second data slice corresponds to, and which part of these untransmitted data the second data slice specifically corresponds to, so as to obtain the sub-slice corresponding to the second data slice.

[0125] In one embodiment, Figure 6 As shown, the above method also includes:

[0126] Step 602, displaying transmission progress information in the display interface, where the transmission progress information is used to indicate the size of data that has been sent to the data receiving end;

[0127] Step 604: when receiving the transmission completion information sent by the data receiving end, update the transmission progress information according to the transmission completion information; the transmission completion information is sent by the data receiving end when the transmission of any first target data segment is completed;

[0128] Step 606: When the transmission suspension event is triggered, the transmission progress information is updated according to the untransmitted data corresponding to each target first data segment.

[0129] In an embodiment of the present application, the transmission progress information may be displayed on the display interface of the data transmitting end. The transmission progress information is used to enable the user to understand the amount of data that has been transmitted to the data receiving end. The transmission progress information may be in the form of a progress bar.

[0130] When the data receiving end verifies that a first data segment has been transmitted in the manner mentioned in the above-mentioned embodiment, the data receiving end may send a transmission completion message to the data sending end. The transmission completion message may carry the identifier of the first data segment. The data sending end updates the transmission progress information according to the ratio of the size of the first data segment to the total size of the target data, so that the ratio of the length of the completed transmission part in the progress bar of the transmission progress information to the length of the entire progress bar is the same as the ratio of the data that has been transmitted to the total size of the target data.

[0131] When a transmission pause event is triggered, the data receiving end can also update the transmission progress information. The data receiving end can update the proportion of the length of the completed transmission part in the progress bar to the entire length of the progress bar according to the size of the untransmitted data, and can also use a special form to display the part of the progress bar that represents the uncompleted transmission (such as highlighting, shadowing, etc.) to indicate that the current state is in a transmission pause state.

[0132] The data transmission method provided in the embodiment of the present application displays the data transmission progress in a visual form on the display interface when the data segment transmission is completed or a transmission suspension event is triggered, so that the user can more clearly understand the amount of data currently completed transmission, thereby improving the user experience.

[0133] In one embodiment, Figure 7 As shown, a data transmission method is provided, comprising:

[0134] S1, the data sending end performs slicing processing on the target data according to a preset slicing size to obtain each first data slicing, and uses a preset number of threads to send each first data slicing in parallel.

[0135] The preset slicing size and the preset number can be configured by those skilled in the art. When slicing the target data according to the preset slicing size, if the length of the last segment of the target data is less than the preset slicing size, the actual length of the last segment of the target data is used as the size of the last first data slicing.

[0136] S2, after any first data fragment is transmitted, the data transmitting end calculates the md5 value of the first data fragment and sends the md5 value to the data receiving end.

[0137] S3, the data receiving end saves the received data to the cache. After any first data fragment is received, the md5 value of the received first data fragment is calculated, and the calculated md5 value is matched with the received md5 value; if the two md5 values ​​do not match, a fragment retransmission request is sent to the data sending end; if they match, the first data fragment is saved from the cache to the local, and a transmission completion message is sent to the data sending end.

[0138] S4, when the data sending end receives a fragment retransmission request, it retransmits the first data fragment, or when it receives transmission completion information, it updates the transmission progress information in the display interface.

[0139] S5, when the transmission suspension event is triggered, the data sending end terminates the transmission of each first data segment and sends thread end notification information to the data receiving end.

[0140] S6, after receiving the thread end notification information, the data receiving end verifies whether the upload of each first data slice is complete, obtains the target first data slice that has not been transmitted and the transmitted data corresponding to each target first data slice, and sends the identifier of the target first data slice and the identifier of the transmitted data to the data sending end.

[0141] S7, when the transmission restart event is triggered, the data sending end re-segmentates the untransmitted data to obtain the second data slices, and respectively determines the slice identification information corresponding to each second data slice, and transmits the second data slices and the slice identification information of each second data slice to the data receiving end in parallel through multiple threads.

[0142] S8, after all second data segments are uploaded, the data sending end initiates a request to verify the integrity of the file, and this request carries the md5 value of the complete file.

[0143] S9, the data receiving end assembles each second data fragment and each first data fragment according to the fragment identification information, calculates the md5 value of the assembled complete file, and compares it with the md5 value sent by the data sending end. If they are consistent, the data receiving end clears the temporary file, keeps the assembled file in the specified directory, completes the file transfer, and encrypts the file as a whole to ensure that the file cannot be used by other systems. The encryption rule is to use an odd number of binary shift digits to achieve it; if they are inconsistent, the data receiving end sends an md5 value sending request to the data sending end, instructing the data sending end to send the md5 values ​​of all fragments. The data receiving end finds the incorrect fragment after comparing the md5 values ​​of the fragments, and returns it to the data sending end, and the data sending end transmits the fragment again.

[0144] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0145] Based on the same inventive concept, the embodiment of the present application also provides a data transmission device for implementing the data transmission method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data transmission device embodiments provided below can refer to the limitations on the data transmission method above, and will not be repeated here.

[0146] In one embodiment, Figure 8 As shown, a data transmission device 800 is provided, comprising: a first transmission module 802, a determination module 804, a fragmentation module 806, and a second transmission module 808, wherein:

[0147] A first transmission module 802 is used to transmit first data slices of target data to a data receiving end in parallel through multiple threads;

[0148] The determination module 804 is used to terminate the transmission of each of the first data slices when the transmission suspension event is triggered, and determine the target first data slices that have not been transmitted in each of the first data slices, and the untransmitted data corresponding to each of the target first data slices;

[0149] The slicing module 806 is used to re-slice each of the untransmitted data to obtain each second data slicing, and respectively determine the slicing identification information corresponding to each of the second data slicing when the transmission restart event is triggered; wherein the slicing identification information at least includes a target first data slicing identification, and the target first data slicing identification is an identification of the target first data slicing to which the data in the second data slicing belongs;

[0150] The second transmission module 808 is used to transmit the second data slices and the slice identification information of the second data slices to the data receiving end in parallel through multiple threads.

[0151] In one embodiment, the determining module 804 is further configured to:

[0152] Sending thread end notification information to the data receiving end;

[0153] receiving target data information sent by the data receiving end in response to the thread termination notification information, the target data information including identifiers of each target first data slice that has not been completely transmitted, and identifiers of transmitted data corresponding to each target first data slice;

[0154] Each of the target first data slices is obtained from the target data information, and the untransmitted data corresponding to each of the target first data slices is determined respectively according to the identifier of the transmitted data corresponding to each of the target first data slices.

[0155] In one embodiment, the slicing module 806 is further configured to:

[0156] Obtaining the number of available threads and network status information, and determining the number of sending threads according to the size of each of the untransmitted data, the number of available threads and the network status information;

[0157] According to the number of sending threads, each of the untransmitted data is re-sliced ​​to obtain each second data slice.

[0158] In one of the embodiments, the network status information includes at least a packet loss rate and a network bandwidth, the number of sending threads is less than or equal to the number of available threads, and the number of sending threads is positively correlated with the size of each untransmitted data and the network bandwidth, and the number of sending threads is negatively correlated with the packet loss rate.

[0159] In one embodiment, the slicing module 806 is further configured to:

[0160] Sequentially concatenate the untransmitted data to obtain target untransmitted data, and equally divide the target untransmitted data into second data slices that are the number of the sending threads.

[0161] In one embodiment, the slicing module 806 is further configured to:

[0162] Determine each sub-slice contained in each second data slice, and determine the first byte offset of each sub-slice corresponding to the target data and the length of each sub-slice, respectively, and the data in the same sub-slice comes from the untransmitted data corresponding to the same target first data slice;

[0163] For any of the second data slices, generate slice identification information corresponding to the second data slice, the slice identification information at least including the first byte offset corresponding to each of the sub-slices, the length of each of the sub-slices, and the identification of the target first data slice corresponding to each of the sub-slices.

[0164] In one embodiment, the slicing module 806 is further configured to:

[0165] respectively determining a second byte offset of each of the untransmitted data in the target untransmitted data, and respectively determining a third byte offset of each of the second data fragments in the target untransmitted data;

[0166] According to the second byte offset and the third byte offset, each sub-shard contained in each second data shard is determined respectively.

[0167] In one embodiment, the device further comprises:

[0168] A display module, used to display transmission progress information in a display interface, wherein the transmission progress information is used to indicate the size of data sent to the data receiving end;

[0169] A first updating module is configured to update the transmission progress information according to the transmission completion information when receiving the transmission completion information sent by the data receiving end; the transmission completion information is sent by the data receiving end when the transmission of any of the first data fragments is completed;

[0170] The second updating module is used to update the transmission progress information according to the untransmitted data corresponding to each of the target first data slices when the transmission suspension event is triggered.

[0171] Each module in the above device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0172] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data transmission method is implemented.

[0173] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0174] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0175] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0176] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0177] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0178] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0179] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0180] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data transmission method, characterized in that: The method comprises: Transmitting each first data slice of the target data to the data receiving end in parallel through multiple threads; In the case of triggering a transmission suspension event, terminating the transmission of each of the first data slices, determining the target first data slices that have not completed transmission in each of the first data slices, and the untransmitted data corresponding to each of the target first data slices; In the case of triggering a restart transmission event, re-slicing each of the untransmitted data to obtain each second data slice, and respectively determining the slice identification information corresponding to each of the second data slices; wherein the slice identification information at least includes a target first data slice identification, and the target first data slice identification is an identification of the target first data slice to which the data in the second data slice belongs; The second data slices and the slice identification information of the second data slices are transmitted to the data receiving end in parallel through multiple threads.

2. The method according to claim 1, characterized in that The determining of the target first data slices that have not been transmitted in each of the first data slices, and the untransmitted data corresponding to each of the target first data slices, includes: Sending thread end notification information to the data receiving end; receiving target data information sent by the data receiving end in response to the thread termination notification information, the target data information including identifiers of each target first data slice that has not been completely transmitted, and identifiers of transmitted data corresponding to each target first data slice; Each of the target first data slices is obtained from the target data information, and the untransmitted data corresponding to each of the target first data slices is determined respectively according to the identifier of the transmitted data corresponding to each of the target first data slices.

3. The method according to claim 1, characterized in that: The re-slicing the untransmitted data to obtain second data slices includes: Obtaining the number of available threads and network status information, and determining the number of sending threads according to the size of each of the untransmitted data, the number of available threads and the network status information; According to the number of sending threads, each of the untransmitted data is re-sliced ​​to obtain each second data slice.

4. The method according to claim 3, characterized in that The network status information includes at least a packet loss rate and a network bandwidth. The number of sending threads is less than or equal to the number of available threads. The number of sending threads is positively correlated with the size of each untransmitted data and the network bandwidth, and the number of sending threads is negatively correlated with the packet loss rate.

5. The method according to claim 3, characterized in that: The re-slicing the untransmitted data according to the number of sending threads to obtain second data slices includes: Sequentially concatenate the untransmitted data to obtain target untransmitted data, and equally divide the target untransmitted data into second data slices that are the number of the sending threads.

6. The method according to claim 5, characterized in that The respectively determining the shard identification information corresponding to each of the second data shards includes: Determine each sub-slice contained in each second data slice, and determine the first byte offset of each sub-slice corresponding to the target data and the length of each sub-slice, respectively, and the data in the same sub-slice comes from the untransmitted data corresponding to the same target first data slice; For any of the second data slices, generate slice identification information corresponding to the second data slice, the slice identification information at least including the first byte offset corresponding to each of the sub-slices, the length of each of the sub-slices, and the identification of the target first data slice corresponding to each of the sub-slices.

7. The method according to claim 6, characterized in that The respectively determining each sub-shard contained in each second data shard comprises: respectively determining a second byte offset of each of the untransmitted data in the target untransmitted data, and respectively determining a third byte offset of each of the second data fragments in the target untransmitted data; According to the second byte offset and the third byte offset, each sub-shard contained in each second data shard is determined respectively.

8. The method according to claim 1, characterized in that The method further comprises: Displaying transmission progress information in a display interface, wherein the transmission progress information is used to indicate the size of data that has been sent to the data receiving end; When receiving the transmission completion information sent by the data receiving end, updating the transmission progress information according to the transmission completion information; the transmission completion information is sent by the data receiving end when the transmission of any of the first data fragments is completed; When the transmission suspension event is triggered, the transmission progress information is updated according to the untransmitted data corresponding to each of the target first data segments.

9. A data transmission device, characterized in that: The device comprises: A first transmission module, used for transmitting each first data slice of the target data to the data receiving end in parallel through multiple threads; A determination module, configured to terminate the transmission of each of the first data slices when a transmission suspension event is triggered, and determine the target first data slices that have not been transmitted in each of the first data slices, and the untransmitted data corresponding to each of the target first data slices; A sharding module, configured to, when a transmission restart event is triggered, re-shard each of the untransmitted data to obtain each second data shard, and respectively determine shard identification information corresponding to each of the second data shards; wherein the shard identification information at least includes a target first data shard identification, and the target first data shard identification is an identification of the target first data shard to which the data in the second data shard belongs; The second transmission module is used to transmit the second data slices and the slice identification information of the second data slices to the data receiving end in parallel through multiple threads.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images