Data transmission method, device, computer equipment, storage medium and computer program product

Abstract

The present application relates to a data transmission method, apparatus, computer equipment, storage medium and computer program product. The method comprises: transmitting each first data slice of target data in parallel; in the case of triggering a transmission pause event, determining the target first data slice that has not completed transmission in the first data slice, and the untransmitted data corresponding to each target first data slice; in the case of triggering a transmission restart event, re-slicing each untransmitted data to obtain each second data slice, and respectively determining the slice identification information corresponding to each second data slice; wherein the slice identification information at least includes the target first data slice identification, and the target first data slice identification is the identification of the target first data slice to which the data in the second data slice belongs; and transmitting each second data slice and the slice identification information of each second data slice in parallel. The use of this method can improve data transmission efficiency.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data transmission method, apparatus, computer equipment, storage medium, and computer program product. Background Art

[0002] With the rapid development of software applications, the need for managing software artifacts is increasing. Artifact management systems have become indispensable in the software development process. Because artifact files are typically large, users often need to pause the upload process to complete other tasks.

[0003] Related technologies typically use multiple threads to transmit finished file segments in parallel. When a user pauses the transfer, they determine which segments have not been fully uploaded and delete the cached files for those segments on the data receiving end. When the user restarts the transfer, the data sending end retransmits the entire incomplete segment. However, due to the large amount of data in each finished file segment, this method of retransmitting the entire segment results in a high amount of data being retransmitted, leading to low data transmission efficiency. Summary of the Invention

[0004] Based on this, it is necessary to provide a data transmission method, apparatus, computer equipment, storage medium and computer program product to address the above technical problems.

[0005] In a first aspect, the present application provides a data transmission method. The method comprises:

[0006] Transmitting the first data fragments of the target data to the data receiving end in parallel through multiple threads;

[0007] When a transmission suspension event is triggered, terminating the transmission of each of the first data slices, determining the target first data slices that have not been completely transmitted in each of the first data slices, and the untransmitted data corresponding to each of the target first data slices;

[0008] In the case of triggering a transmission restart event, re-sharding each of the untransmitted data to obtain each second data shard, and respectively determining shard identification information corresponding to each of the second data shards; wherein the shard identification information at least includes a target first data shard identification, and the target first data shard identification is an identification of the target first data shard to which the data in the second data shard belongs;

[0009] The second data slices and the slice identification information of the second data slices are transmitted to the data receiving end in parallel through multiple threads.

[0010] In one embodiment, determining the target first data slices that have not been transmitted in each of the first data slices and the untransmitted data corresponding to each of the target first data slices includes:

[0011] Sending thread end notification information to the data receiving end;

[0012] receiving target data information sent by the data receiving end in response to the thread termination notification information, the target data information including an identifier of each target first data fragment that has not been completely transmitted, and an identifier of transmitted data corresponding to each target first data fragment;

[0013] Each of the target first data slices is obtained from the target data information, and the untransmitted data corresponding to each of the target first data slices is determined according to an identifier of the transmitted data corresponding to each of the target first data slices.

[0014] In one embodiment, the re-slicing each of the untransmitted data to obtain each second data slice includes:

[0015] Obtaining the number of available threads and network status information, and determining the number of sending threads according to the size of each untransmitted data, the number of available threads, and the network status information;

[0016] The untransmitted data are re-sliced according to the number of sending threads to obtain second data slices.

[0017] In one embodiment, the network status information includes at least packet loss rate and network bandwidth, the number of sending threads is less than or equal to the number of available threads, and the number of sending threads is positively correlated with the size of each untransmitted data and the network bandwidth, and the number of sending threads is negatively correlated with the packet loss rate.

[0018] In one embodiment, the re-slicing of each of the untransmitted data according to the number of sending threads to obtain each second data slice includes:

[0019] Sequential splicing processing is performed on each of the untransmitted data to obtain target untransmitted data, and the target untransmitted data is equally divided into second data slices that are the number of the sending threads.

[0020] In one embodiment, the determining the shard identification information corresponding to each of the second data shards includes:

[0021] Determine each sub-slice contained in each second data slice, and determine the first byte offset of each sub-slice corresponding to the target data and the length of each sub-slice, wherein the data in the same sub-slice comes from the untransmitted data corresponding to the same target first data slice;

[0022] For any of the second data shards, generate shard identification information corresponding to the second data shard, the shard identification information at least including the first byte offset corresponding to each of the sub-shards, the length of each of the sub-shards, and the identification of the target first data shard corresponding to each of the sub-shards.

[0023] In one embodiment, the step of respectively determining each sub-shard contained in each second data shard includes:

[0024] respectively determining a second byte offset of each of the untransmitted data in the target untransmitted data, and respectively determining a third byte offset of each of the second data fragments in the target untransmitted data;

[0025] According to the second byte offset and the third byte offset, each sub-shard contained in each second data shard is determined respectively.

[0026] In one embodiment, the method further comprises:

[0027] Displaying transmission progress information on a display interface, wherein the transmission progress information is used to indicate the size of data sent to the data receiving end;

[0028] Upon receiving transmission completion information sent by the data receiving end, updating the transmission progress information according to the transmission completion information; the transmission completion information is sent by the data receiving end when the transmission of any of the first data fragments is completed;

[0029] When the transmission suspension event is triggered, the transmission progress information is updated according to the untransmitted data corresponding to each of the target first data segments.

[0030] In a second aspect, the present application further provides a data transmission device. The device comprises:

[0031] A first transmission module, configured to transmit, in parallel, first data fragments of target data to a data receiving end through multiple threads;

[0032] a determination module, configured to, when a transmission suspension event is triggered, terminate the transmission of each of the first data slices, determine target first data slices for which transmission has not been completed in each of the first data slices, and untransmitted data corresponding to each of the target first data slices;

[0033] a sharding module configured to, upon triggering a transmission restart event, re-shard each of the untransmitted data to obtain each second data shard, and respectively determine shard identification information corresponding to each of the second data shards; wherein the shard identification information includes at least a target first data shard identification, the target first data shard identification being an identification of the target first data shard to which the data in the second data shard belongs;

[0034] The second transmission module is configured to transmit the second data slices and the slice identification information of the second data slices to the data receiving end in parallel through multiple threads.

[0035] In one embodiment, the determining module is further configured to:

[0036] Sending thread end notification information to the data receiving end;

[0037] receiving target data information sent by the data receiving end in response to the thread termination notification information, the target data information including an identifier of each target first data fragment that has not been completely transmitted, and an identifier of transmitted data corresponding to each target first data fragment;

[0038] Each of the target first data slices is obtained from the target data information, and the untransmitted data corresponding to each of the target first data slices is determined according to an identifier of the transmitted data corresponding to each of the target first data slices.

[0039] In one embodiment, the sharding module is further configured to:

[0040] Obtaining the number of available threads and network status information, and determining the number of sending threads according to the size of each untransmitted data, the number of available threads, and the network status information;

[0041] The untransmitted data are re-sliced according to the number of sending threads to obtain second data slices.

[0042] In one embodiment, the network status information includes at least packet loss rate and network bandwidth, the number of sending threads is less than or equal to the number of available threads, and the number of sending threads is positively correlated with the size of each untransmitted data and the network bandwidth, and the number of sending threads is negatively correlated with the packet loss rate.

[0043] In one embodiment, the sharding module is further configured to:

[0044] Sequential splicing processing is performed on each of the untransmitted data to obtain target untransmitted data, and the target untransmitted data is equally divided into second data slices that are the number of the sending threads.

[0045] In one embodiment, the sharding module is further configured to:

[0046] Determine each sub-slice contained in each second data slice, and determine the first byte offset of each sub-slice corresponding to the target data and the length of each sub-slice, wherein the data in the same sub-slice comes from the untransmitted data corresponding to the same target first data slice;

[0047] For any of the second data shards, generate shard identification information corresponding to the second data shard, the shard identification information at least including the first byte offset corresponding to each of the sub-shards, the length of each of the sub-shards, and the identification of the target first data shard corresponding to each of the sub-shards.

[0048] In one embodiment, the sharding module is further configured to:

[0049] respectively determining a second byte offset of each of the untransmitted data in the target untransmitted data, and respectively determining a third byte offset of each of the second data fragments in the target untransmitted data;

[0050] According to the second byte offset and the third byte offset, each sub-shard contained in each second data shard is determined respectively.

[0051] In one embodiment, the apparatus further comprises:

[0052] A display module, configured to display transmission progress information on a display interface, wherein the transmission progress information is used to indicate the size of data sent to the data receiving end;

[0053] a first updating module configured to update the transmission progress information according to the transmission completion information when receiving the transmission completion information sent by the data receiving end; the transmission completion information is sent by the data receiving end when the transmission of any of the first data fragments is completed;

[0054] The second updating module is configured to update the transmission progress information according to the untransmitted data corresponding to each of the target first data slices when the transmission suspension event is triggered.

[0055] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above methods when executing the computer program.

[0056] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements any of the above methods when executed by a processor.

[0057] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which implements any of the above methods when executed by a processor.

[0058] The above-mentioned data transmission method, apparatus, computer equipment, storage medium, and computer program product, after pausing data transmission, obtains each target first data slice that has not been fully transmitted, and determines the untransmitted data from each target first data slice. After restarting data transmission, the data sending end further splits the untransmitted data into second data slices, and generates slice identification information for indicating which target first data slices the data in each second data slice originally belonged to. The second data slice and the slice identification information are transmitted together to the data receiving end, so that the data receiving end can refer to the slice identification information and correctly splice the data in the second data slice with the data already received in the original target first data slice. Therefore, the amount of data that needs to be retransmitted each time the transmission is restarted can be reduced, while also ensuring that the retransmitted data is correctly spliced with the transmitted data, thereby improving data transmission efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 1 is a flow chart of a data transmission method according to an embodiment;

[0060] Figure 2 104 is a flow chart of step 104 in one embodiment;

[0061] Figure 3 106 is a flow chart of step 106 in one embodiment;

[0062] Figure 4 106 is a flow chart of step 106 in one embodiment;

[0063] Figure 5 4 is a flow chart of step 402 in one embodiment;

[0064] Figure 6 A schematic diagram of a process for displaying transmission progress information in one embodiment;

[0065] Figure 7 1 is a flow chart of a data transmission method according to an embodiment;

[0066] Figure 8 is a structural block diagram of a data transmission device in one embodiment;

[0067] Figure 9FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0069] In one embodiment, Figure 1 As shown, a data transmission method is provided. This embodiment uses the method applied to a data sending end as an example for illustration, and includes the following steps:

[0070] Step 102: Transmitting each first data fragment of the target data to the data receiving end in parallel through multiple threads.

[0071] In the embodiments of the present application, the target data is a file that needs to be uploaded to the backend (data receiving end). When sending the target data, the data sending end first splits the target data into multiple first data slices. This application does not specifically limit the method for splitting the target data. For example, the data slice size can be pre-set, and the target data can be split into multiple first data slices according to the data slice size, so that the size of each first data slice is the pre-set data slice size; alternatively, the number of slices can be pre-set, and the target data can be split into a pre-set number of first data slices, etc.

[0072] After obtaining each first data slice, the data transmitter transmits each first data slice in parallel to the data receiver using multiple threads. If the number of first data slices does not exceed the number of available threads, the data transmitter begins transmitting all first data slices simultaneously. If the number of first data slices exceeds the number of available threads, the data transmitter transmits each first data slice to the data receiver in multiple rounds, transmitting a portion of each first data slice in each round.

[0073] Step 104 , when the transmission suspension event is triggered, terminate the transmission of each first target data slice, determine the target first data slices that have not been completely transmitted in each first data slice, and determine the untransmitted data corresponding to each target first data slice.

[0074] In an embodiment of the present application, the transmission pause event can be an event detected by the data sending end, such as a user triggering a transmission pause control on a display interface of the data sending end. The transmission pause event can also be an event detected by the data receiving end. For example, when the data receiving end determines that the current network quality is poor, resulting in poor transmission quality of each first data fragment (e.g., a high packet loss rate, slow transmission speed, etc.), it can determine to trigger the transmission pause event and notify the data sending end of the occurrence of the transmission pause event.

[0075] When a transmission suspension event is triggered, the data sending end stops transmitting each first data slice through each thread, and determines the target first data slice that has not been transmitted and the untransmitted data in each target first data slice that has not been transmitted to the data receiving end.

[0076] When transmitting the first data segment, the data sending end can record in real time which part of the first data segment is currently being transmitted. For example, when sending a data packet of the first data segment, the data sending end can determine the byte offset of the data contained in the data packet in the first data segment. When a transmission pause event is triggered, the data sending end can obtain each first data segment currently being transmitted, and determine whether the first data segment has been completed based on the total number of bytes of the first data segment and the byte offset recorded in the last data packet sent before the transmission pause event is triggered, and if not, which part has not been completed.

[0077] If the data sending end transmits the first data slices to the data receiving end in multiple rounds, the data sending end, in addition to determining the target first data slices that have not yet been transmitted from the first data slices currently being transmitted using the above method, also needs to consider each first data slice that has not yet started transmission as the target first data slice. The untransmitted data corresponding to each first data slice that has not yet started transmission is the entire data in these first data slices.

[0078] Step 106: When a transmission restart event is triggered, each untransmitted data is re-sharded to obtain each second data shard, and the shard identification information corresponding to each second data shard is determined respectively; wherein the shard identification information at least includes the target first data shard identification, and the target first data shard identification is the identification of the target first data shard to which the data in the second data shard belongs.

[0079] In the embodiments of the present application, a transmission restart event is an event that is the opposite of a transmission pause event. For example, if the transmission pause event is caused by a user triggering a transmission pause control on the display interface of the data transmitting end, the transmission restart event is caused by the user triggering a transmission restart control on the display interface of the data transmitting end. If the transmission pause event is caused by poor network quality, the transmission restart event may be caused by the data receiving end detecting that the network quality has improved. For example, when the data receiving end is communicating with another data transmitting end, the data receiving end detects that the packet loss rate or data transmission speed meets the requirements.

[0080] After the transmission restart event is triggered, the data transmitter re-shards the untransmitted data corresponding to each target first data shard. The number of shards during re-sharding can be determined based on the current network status, the current number of available threads, etc.

[0081] Resharding can be performed by sharding each untransmitted data separately. That is, when the target first data shards are shards A and B, the shards obtained after resharding can be shards A1, A2, B1, and B2, where the data in A1 and A2 all come from A, and the data in B1 and B2 all come from B. Resharding can also be performed by mixing the untransmitted data and then sharding them together. That is, when the target first data shards are shards A and B, the shards obtained after resharding can be shards C1 and C2. The data in C1 may come from both A and B, and the data in C2 may also come from both A and B.

[0082] After resharding to obtain second data slices, it is necessary to generate slice identification information for each second data slice. The slice identification information includes at least the identifier of each target first data slice, which indicates which portion of the data in each second data slice originates from which target first data slice. This allows the data receiving end to read the slice identification information and correctly combine the data in the second data slice with the previously transmitted data in the target first data slice received by the data receiving end to obtain the complete target data.

[0083] It should be noted that if the re-sharding strategy adopted will fragment the untransmitted data from the same target first data fragment into different second data fragments, the fragment identification information may also include a splicing order identifier for each target first data fragment identifier, and the splicing order identifier is used to indicate that the untransmitted data in the target first data fragment is specifically divided into several second data fragments, and when the data corresponding to the target first data fragment in each second data fragment is spliced to obtain the untransmitted data of the target first data fragment, the data in the second data fragment corresponding to the target first data fragment has the corresponding splicing order in the splicing.

[0084] For example, if the untransmitted data of the target first data shard A is sequentially split into A1, A2, and A3, and A1, A2, and A3 belong to different second data shards respectively, then for the second data shard where A1 is located, the shard identification information corresponding to the second data shard should record the identifier of the target first data shard A (assuming A is used as the identifier), and the splicing order identifier for identifier A (recording that identifier A corresponds to 3 second data shards, and the splicing order corresponding to this second data shard is 1). In this way, after the data receiving end reads the shard identification information of each second data shard, if it is necessary to obtain the untransmitted data of the target first data shard A, the data receiving end can obtain the second data shards containing identifier A in the shard identification information of each second data shard, and sequentially splice the data corresponding to identifier A in these second data shards according to the splicing order identifiers in these second data shards to obtain the untransmitted data of the target first data shard A.

[0085] Step 108: Transmit each second data slice and the slice identification information of each second data slice to the data receiving end in parallel through multiple threads.

[0086] In an embodiment of the present application, the data transmitting end transmits each second data fragment and the fragment identification information of each second data fragment in parallel to the data receiving end. The data transmitting end may preferentially send the fragment identification information of each second data fragment to the data receiving end before sending the second data fragment, so that the data receiving end can refer to the fragment identification information in real time during the process of receiving each second data fragment and combine the received data with the transmitted data cached in the data receiving end.

[0087] It should be noted that when the data transmitter transmits each first data fragment, it may also transmit characteristic information of each first data fragment to the data receiver. This characteristic information is used to verify the integrity of the first data fragment. For example, the characteristic information may be the MD5 (Message-Digest Algorithm 5) value of the first data fragment or any digital signature. After determining that the transmission of the first data fragment is complete, the data receiver may calculate the characteristic information of the received first data fragment and compare the calculated characteristic information with the characteristic information sent by the data transmitter to determine whether the data receiver has received the entire first data fragment.

[0088] After parsing each second data fragment and combining the data in each second data fragment with the transmitted data of the target first data fragment to which it originally belonged, the data receiving end may also calculate characteristic information of the combined data and compare the calculated characteristic information with the characteristic information sent by the data sending end for the target first data fragment to determine whether the target first data fragment received by the data receiving end is complete. If it is incomplete, the data receiving end may notify the data sending end to resend the target first data fragment, or may notify the data sending end to resend each second data fragment carrying the untransmitted data of the target first data fragment.

[0089] The data transmission method provided in the embodiment of the present application obtains each target first data fragment that has not been fully transmitted after pausing data transmission, and determines the untransmitted data from each target first data fragment. After restarting data transmission, the data sending end further splits the untransmitted data into second data fragments, and generates fragment identification information for indicating which target first data fragments the data in each second data fragment originally belonged to, and transmits the second data fragment and the fragment identification information together to the data receiving end, so that the data receiving end can refer to the fragment identification information and correctly splice the data in the second data fragment with the data already received in the original target first data fragment. Therefore, the amount of data that needs to be retransmitted each time the transmission is restarted can be reduced, while also ensuring that the retransmitted data and the transmitted data are correctly spliced, thereby improving data transmission efficiency.

[0090] In one embodiment, Figure 2 As shown, in step 104, determining the target first data slices that have not been transmitted in each first data slice and the untransmitted data corresponding to each target first data slice includes:

[0091] Step 202: Send thread termination notification information to the data receiving end.

[0092] Step 204: Receive target data information sent by the data receiving end in response to the thread termination notification information. The target data information includes identifiers of each target first data segment that has not been completely transmitted and identifiers of transmitted data corresponding to each target first data segment.

[0093] Step 206 : Acquire each target first data slice from the target data information, and determine the untransmitted data corresponding to each target first data slice according to the identifier of the transmitted data corresponding to each target first data slice.

[0094] In the embodiment of the present application, considering that in the event of network fluctuations and other issues, the data sent by the data sending end may not necessarily reach the data receiving end, and therefore the determination by the data sending end as to which data is not transmitted may be inaccurate. The data sending end may instruct the data receiving end to determine the target first data slice and which data in the target first data slice is not transmitted.

[0095] When a transmission pause event is triggered, the data sending end sends a thread termination notification message to the data receiving end to inform the data receiving end that the data sending end has terminated each data transmission thread. After receiving the thread termination notification message, the data receiving end determines which first data segments have completed transmission.

[0096] The embodiment of the present application does not specifically limit how the data receiving end determines which first data fragments have completed transmission. The specific determination method can be set according to the data transmission format agreed upon between the data sending end and the data receiving end. For example, in each data packet sent by the data sending end to the data receiving end, it carries the total number of data packets corresponding to the first data fragment and the number of the currently transmitted data packet in the total number. The data receiving end can determine whether the first data fragment has been transmitted based on whether the number of the last data packet received before receiving the thread end notification information is equal to the total number of data packets. Or in the case where the data sending end has sent the characteristic information of the first data fragment to the data receiving end, the data receiving end can calculate the characteristic information of each data currently received for the first data fragment, and determine whether the first data fragment has been transmitted based on whether the characteristic information matches the characteristic information sent by the data sending end.

[0097] After the data receiving end determines the target first data segments that have not yet been transmitted, it obtains identifiers for these target first data segments. The data receiving end also needs to determine identifiers that can be used to identify which data within these target first data segments is the transmitted data. For example, the data receiving end can calculate the number of bytes of received data and use this number as the identifier of the transmitted data. Alternatively, if the data packets carry numbers, the data receiving end can use the numbers of each received data packet as the identifier of the transmitted data.

[0098] The data receiving end then sends the identifier of the target first data fragment and the identifier of the transmitted data as target data information to the data sending end.

[0099] After receiving the target data information, the data transmitter determines the target first data fragment based on the identifier of the target first data fragment. The data transmitter can also determine untransmitted data based on the identifier of the transmitted data. For example, if the identifier of the transmitted data is a number of bytes, the data transmitter can define the data starting from the start position of the first data fragment and up to that number of bytes as the transmitted data, and define the data other than the transmitted data as the untransmitted data. If the identifier of the transmitted data is the number of each received data packet, the data transmitter can determine the number of each untransmitted data packet based on this information and define the data in these data packets as the untransmitted data.

[0100] The data transmission method provided by the embodiment of the present application enables the data receiving end to determine the target first data fragments that have not been transmitted, and determines the transmitted data in each target first data fragment, so that the data sending end can determine the untransmitted data based on the transmitted data. Determining the untransmitted data based on the data actually received by the data receiving end can reduce the data sent by the data sending end due to network fluctuations, which is different from the data received by the data receiving end, causing the data sending end to misjudge the untransmitted data, resulting in the data receiving end failing to verify the integrity of the target data, and requiring the data sending end to retransmit the data. This can further improve the efficiency of data transmission.

[0101] In one embodiment, Figure 3 As shown, in step 106, each untransmitted data is re-sliced to obtain each second data slice, including:

[0102] Step 302: Obtain the number of available threads and network status information, and determine the number of sending threads according to the size of each untransmitted data, the number of available threads, and the network status information.

[0103] Step 304: Re-segment each untransmitted data according to the number of sending threads to obtain each second data segment.

[0104] In an embodiment of the present application, when re-sharding, the data sending end can determine the number of sending threads based on the currently available number of threads, the size of the untransmitted data and the network status, and obtain each second data fragment by fragmenting according to the number of sending threads, so that the data sending end can transmit all the second data fragments in parallel at the same time through the sending number of threads, thereby improving the efficiency of data transmission.

[0105] The total size of the untransmitted data can be calculated based on the sum of the sizes of the individual untransmitted data. The number of sending threads can be positively correlated with the total size, so that when the untransmitted data is large, the data sending end can use more threads to simultaneously send each second data fragment, thereby improving data transmission efficiency.

[0106] The network status may include at least packet loss rate and network bandwidth. The packet loss rate may be negatively correlated with the number of sending threads. When the packet loss rate is high, the probability that the data sender needs to process retransmission requests sent by the data receiver when packet loss is detected is also high. Therefore, the data sender may determine a smaller number of sending threads to allocate some resources to process retransmission requests and avoid overloading the data sender.

[0107] The network bandwidth can be positively correlated with the number of sending threads. When the bandwidth is low, the data sender can also determine a smaller number of sending threads to avoid excessive amounts of data being transmitted simultaneously, which may cause network conditions to deteriorate.

[0108] The number of available threads is determined by the current load on the data sender. Based on the current load, the data sender can determine the maximum amount of resources it can allocate to send untransmitted data. Based on the allocated resources, the maximum number of threads to use for sending, also known as the number of available threads, should be less than or equal to the number of available threads.

[0109] The specific number of available threads can be determined by those skilled in the art according to the above principles and actual needs.

[0110] The data transmission method provided in the embodiment of the present application determines the number of sending threads based on the number of available threads and network status information, and fragments data according to the number of sending threads. Each fragment can be sent simultaneously using a number of threads suitable for the current network status, thereby improving data transmission efficiency.

[0111] In one embodiment, each untransmitted data is re-sliced according to the number of sending threads to obtain each second data slice, including:

[0112] Sequential splicing processing is performed on each untransmitted data to obtain target untransmitted data, and the target untransmitted data is equally divided into second data slices that are the number of sending threads.

[0113] In an embodiment of the present application, the target untransmitted data is obtained by sequentially concatenating the untransmitted data. The target untransmitted data is then divided equally according to the number of sending threads to obtain the respective second data slices. This method ensures that the size of each second data slice is approximately the same, so that when the second data slices are sent in parallel, each second data slice can be transmitted in approximately the same time. Furthermore, the untransmitted data of each target first data slice can be dispersed as much as possible into a smaller number of second data slices, making it easier for the data receiving end to extract the untransmitted data from the second data slices.

[0114] In one embodiment, Figure 4 As shown, in step 106, the shard identification information corresponding to each second data shard is determined respectively, including:

[0115] Step 402: Determine each sub-slice contained in each second data slice, and determine the first byte offset of the target data corresponding to each sub-slice and the length of each sub-slice. The data in the same sub-slice comes from the untransmitted data corresponding to the same target first data slice.

[0116] Step 404: For any second data slice, generate slice identification information corresponding to the second data slice, where the slice identification information at least includes the first byte offset corresponding to each sub-slice, the length of each sub-slice, and the identifier of the target first data slice corresponding to each sub-slice.

[0117] In an embodiment of the present application, when determining the shard identification information corresponding to the second data shard, the sub-shards contained in the second data shard can be first determined. A sub-shard refers to the portion of the second data shard that corresponds to the same target first data shard. Because each second data shard is obtained by sequentially concatenating and segmenting the untransmitted data of each target first data shard, the portion of the second data shard that corresponds to the same target first data shard will be a portion of continuous data.

[0118] When determining the untransmitted data for each target first data slice, the data receiving end may simultaneously determine the byte offset of the untransmitted data relative to the target data, where the byte offset indicates the byte number from the beginning of the target data at which the untransmitted data begins. When segmenting the second data slice, the data receiving end may further determine the first byte offset of each sub-slice in the second data slice relative to the target data based on the byte from the beginning of the untransmitted data at which the segmentation point is located.

[0119] Slice identification information can be generated based on the first byte offset corresponding to each sub-slice and the length of each sub-slice. The slice identification information includes the identifier of the target first data slice corresponding to each sub-slice, the length of each sub-slice, and the first byte offset of each sub-slice relative to the target data. This allows the data receiving end to read the complete sub-slice from the second data slice based on the length of the sub-slice, determine which transmitted data to combine the sub-slice with based on the identifier of the target first data slice corresponding to each sub-slice, and determine the order in which to combine the sub-slices based on the first byte offset of each sub-slice.

[0120] In one embodiment, Figure 5 As shown, in step 402, each sub-shard contained in each second data shard is determined, including:

[0121] Step 502, respectively determining the second byte offset of each untransmitted data in the target untransmitted data, and respectively determining the third byte offset of each second data fragment in the target untransmitted data;

[0122] Step 504: Determine each sub-shard contained in each second data shard according to the second byte offset and the third byte offset.

[0123] In this embodiment of the present application, the second byte offset of the untransmitted data in the target untransmitted data is used to indicate the byte number from the beginning of the target untransmitted data at which the untransmitted data begins. The third byte offset of the second data fragment in the target untransmitted data is used to indicate the byte number from the beginning of the target untransmitted data at which the second data fragment begins.

[0124] For a particular second data slice, based on the third byte offset of the second data slice and the third byte offset of the second data slice next to the second data slice, the byte range covered by the second data slice in the target untransmitted data can be determined. Similarly, the byte range covered by each untransmitted data can also be determined. For a particular second data slice, based on the byte range covered by the untransmitted data that overlaps with the byte range covered by the second data slice, it can be determined which untransmitted data the second data slice corresponds to, and to which part of the untransmitted data the second data slice specifically corresponds, thereby obtaining the sub-slice corresponding to the second data slice.

[0125] In one embodiment, Figure 6 As shown, the above method also includes:

[0126] Step 602: Displaying transmission progress information on the display interface, where the transmission progress information is used to indicate the size of data sent to the data receiving end;

[0127] Step 604: upon receiving a transmission completion message sent by the data receiving end, updating the transmission progress information according to the transmission completion message; the transmission completion message is sent by the data receiving end upon completion of transmission of any first target data segment;

[0128] Step 606: When the transmission suspension event is triggered, the transmission progress information is updated according to the untransmitted data corresponding to each target first data segment.

[0129] In an embodiment of the present application, transmission progress information can be displayed on the display interface of the data transmitting end. The transmission progress information is used to allow the user to understand the amount of data transmitted to the data receiving end. The transmission progress information can be in the form of a progress bar.

[0130] When the data receiving end verifies that a first data segment has been transmitted completely using the methods described in the aforementioned embodiments, it may send a transmission completion message to the data sending end. The transmission completion message may include an identifier for the first data segment. The data sending end updates the transmission progress information based on the ratio of the size of the first data segment to the total size of the target data, such that the ratio of the length of the completed transmission portion in the progress bar of the transmission progress information to the total length of the progress bar is the same as the ratio of the completed data to the total size of the target data.

[0131] When a transfer pause event is triggered, the data receiver can also update the transfer progress information. Based on the size of the untransmitted data, the data receiver can update the ratio of the length of the progress bar representing the completed transfer to the total length of the progress bar. The data receiver can also use a special display format (such as highlighting or shading) to indicate that the transfer is currently paused.

[0132] The data transmission method provided in the embodiment of the present application displays the data transmission progress in a visual form on the display interface when the data segment transmission is completed or a transmission suspension event is triggered, so that the user can more clearly understand the amount of data currently completed transmission and improve the user experience.

[0133] In one embodiment, Figure 7 As shown, a data transmission method is provided, comprising:

[0134] S1. The data sending end slices the target data according to a preset slice size to obtain first data slices, and uses a preset number of threads to send the first data slices in parallel.

[0135] The preset fragment size and the preset number can be configured by those skilled in the art. When fragmenting the target data according to the preset fragment size, if the length of the last segment of the target data is less than the preset fragment size, the actual length of the last segment of the target data is used as the size of the last first data fragment.

[0136] S2. After any first data fragment is transmitted, the data transmitting end calculates the MD5 value of the first data fragment and sends the MD5 value to the data receiving end.

[0137] S3, the data receiving end saves the received data to the cache. After any first data fragment is received, the MD5 value of the received first data fragment is calculated, and the calculated MD5 value is matched with the received MD5 value. If the two MD5 values do not match, a fragment retransmission request is sent to the data sending end. If they match, the first data fragment is saved from the cache to the local, and a transmission completion message is sent to the data sending end.

[0138] S4, when the data sending end receives the fragment retransmission request, it retransmits the first data fragment, or when it receives the transmission completion information, it updates the transmission progress information in the display interface.

[0139] S5. When the transmission suspension event is triggered, the data sending end terminates the transmission of each first data segment and sends thread termination notification information to the data receiving end.

[0140] S6. After receiving the thread end notification information, the data receiving end verifies whether the upload of each first data slice is complete, obtains the target first data slice that has not been completed and the transmitted data corresponding to each target first data slice, and sends the identifier of the target first data slice and the identifier of the transmitted data to the data sending end.

[0141] S7, when the data sending end triggers the restart transmission event, it re-shards the untransmitted data to obtain each second data fragment, and determines the fragment identification information corresponding to each second data fragment, and transmits each second data fragment and the fragment identification information of each second data fragment to the data receiving end in parallel through multiple threads.

[0142] S8: After all second data segments are uploaded, the data sending end initiates a request to verify the integrity of the file. This request carries the MD5 value of the complete file.

[0143] S9: The data receiving end assembles each second data fragment with each first data fragment based on the fragment identification information, calculates the MD5 value of the assembled complete file, and compares it with the MD5 value sent by the data sending end. If they are consistent, the data receiving end clears the temporary file, retains the assembled file in the designated directory, completes the file transfer, and encrypts the entire file to ensure that the file cannot be used by other systems. The encryption rule is to use binary shifting of an odd number of digits. If they are inconsistent, the data receiving end sends an MD5 value sending request to the data sending end, instructing the data sending end to send the MD5 values of all fragments. The data receiving end compares the MD5 values of the fragments and finds the incorrect fragment, returns it to the data sending end, and the data sending end transmits the incorrect fragment again.

[0144] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0145] Based on the same inventive concept, embodiments of the present application further provide a data transmission device for implementing the aforementioned data transmission method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of one or more data transmission device embodiments provided below can be found in the above-mentioned limitations of the data transmission method and will not be further elaborated here.

[0146] In one embodiment, Figure 8 As shown, a data transmission device 800 is provided, comprising: a first transmission module 802, a determination module 804, a fragmentation module 806, and a second transmission module 808, wherein:

[0147] A first transmission module 802 is configured to transmit, in parallel, first data fragments of target data to a data receiving end through multiple threads;

[0148] a determination module 804 configured to, when a transmission suspension event is triggered, terminate the transmission of each of the first data slices, determine target first data slices for which transmission has not been completed in each of the first data slices, and untransmitted data corresponding to each of the target first data slices;

[0149] a fragmentation module 806 configured to, upon triggering a transmission restart event, re-fragment each of the untransmitted data to obtain each second data fragment, and to determine fragment identification information corresponding to each of the second data fragments; wherein the fragment identification information includes at least a target first data fragment identification, the target first data fragment identification being an identification of the target first data fragment to which the data in the second data fragment belongs;

[0150] The second transmission module 808 is configured to transmit the second data slices and the slice identification information of the second data slices to the data receiving end in parallel through multiple threads.

[0151] In one embodiment, the determining module 804 is further configured to:

[0152] Sending thread end notification information to the data receiving end;

[0153] receiving target data information sent by the data receiving end in response to the thread termination notification information, the target data information including an identifier of each target first data fragment that has not been completely transmitted, and an identifier of transmitted data corresponding to each target first data fragment;

[0154] Each of the target first data slices is obtained from the target data information, and the untransmitted data corresponding to each of the target first data slices is determined according to an identifier of the transmitted data corresponding to each of the target first data slices.

[0155] In one embodiment, the sharding module 806 is further configured to:

[0156] Obtaining the number of available threads and network status information, and determining the number of sending threads according to the size of each untransmitted data, the number of available threads, and the network status information;

[0157] The untransmitted data are re-sliced according to the number of sending threads to obtain second data slices.

[0158] In one embodiment, the network status information includes at least packet loss rate and network bandwidth, the number of sending threads is less than or equal to the number of available threads, and the number of sending threads is positively correlated with the size of each untransmitted data and the network bandwidth, and the number of sending threads is negatively correlated with the packet loss rate.

[0159] In one embodiment, the sharding module 806 is further configured to:

[0160] Sequential splicing processing is performed on each of the untransmitted data to obtain target untransmitted data, and the target untransmitted data is equally divided into second data slices that are the number of the sending threads.

[0161] In one embodiment, the sharding module 806 is further configured to:

[0162] Determine each sub-slice contained in each second data slice, and determine the first byte offset of each sub-slice corresponding to the target data and the length of each sub-slice, wherein the data in the same sub-slice comes from the untransmitted data corresponding to the same target first data slice;

[0163] For any of the second data shards, generate shard identification information corresponding to the second data shard, the shard identification information at least including the first byte offset corresponding to each of the sub-shards, the length of each of the sub-shards, and the identification of the target first data shard corresponding to each of the sub-shards.

[0164] In one embodiment, the sharding module 806 is further configured to:

[0165] respectively determining a second byte offset of each of the untransmitted data in the target untransmitted data, and respectively determining a third byte offset of each of the second data fragments in the target untransmitted data;

[0166] According to the second byte offset and the third byte offset, each sub-shard contained in each second data shard is determined respectively.

[0167] In one embodiment, the apparatus further comprises:

[0168] A display module, configured to display transmission progress information on a display interface, wherein the transmission progress information is used to indicate the size of data sent to the data receiving end;

[0169] a first updating module configured to update the transmission progress information according to the transmission completion information when receiving the transmission completion information sent by the data receiving end; the transmission completion information is sent by the data receiving end when the transmission of any of the first data fragments is completed;

[0170] The second updating module is configured to update the transmission progress information according to the untransmitted data corresponding to each of the target first data slices when the transmission suspension event is triggered.

[0171] Each module in the above-mentioned apparatus may be implemented in whole or in part by software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0172] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 9 As shown. The computer device includes a processor, memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a data transmission method.

[0173] Those skilled in the art will understand that Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0174] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0175] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0176] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0177] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0178] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0179] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0180] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data transmission method, characterized in that: The method comprises: Transmitting the first data fragments of the target data to the data receiving end in parallel through multiple threads; When a transmission suspension event is triggered, terminating the transmission of each of the first data slices, determining the target first data slices that have not been completely transmitted in each of the first data slices, and the untransmitted data corresponding to each of the target first data slices; In the case of triggering a restart transmission event, the untransmitted data are mixed and then re-sharded to obtain each second data slice, and the slice identification information corresponding to each second data slice is determined respectively; wherein the slice identification information includes at least a target first data slice identifier and a splicing order identifier of the target first data slice identifier, the target first data slice identifier is an identifier of the target first data slice to which the data in the second data slice belongs, and the splicing order identifier is used to indicate each second data slice to which the untransmitted data in the target first data slice belongs, as well as the corresponding splicing order of the data in the second data slice corresponding to the target first data slice when the target first data slice is spliced to be obtained; The second data slices and the slice identification information of the second data slices are transmitted to the data receiving end in parallel through multiple threads.

2. The method according to claim 1, characterized in that The determining of target first data slices that have not been transmitted in each of the first data slices and untransmitted data corresponding to each of the target first data slices includes: Sending thread end notification information to the data receiving end; receiving target data information sent by the data receiving end in response to the thread termination notification information, the target data information including an identifier of each target first data fragment that has not been completely transmitted, and an identifier of transmitted data corresponding to each target first data fragment; Each of the target first data slices is obtained from the target data information, and the untransmitted data corresponding to each of the target first data slices is determined according to an identifier of the transmitted data corresponding to each of the target first data slices.

3. The method according to claim 1, characterized in that The mixing of the untransmitted data and re-slicing to obtain second data slices includes: Obtaining the number of available threads and network status information, and determining the number of sending threads according to the size of each untransmitted data, the number of available threads, and the network status information; According to the number of sending threads, the untransmitted data are mixed and then re-sharded to obtain second data fragments.

4. The method according to claim 3, characterized in that The network status information includes at least a packet loss rate and a network bandwidth. The number of sending threads is less than or equal to the number of available threads. The number of sending threads is positively correlated with the size of each untransmitted data and the network bandwidth, and negatively correlated with the packet loss rate.

5. The method according to claim 3, characterized in that The re-slicing of each of the untransmitted data according to the number of sending threads to obtain each second data slice includes: Sequential splicing processing is performed on each of the untransmitted data to obtain target untransmitted data, and the target untransmitted data is equally divided into second data slices that are the number of the sending threads.

6. The method according to claim 5, characterized in that The respectively determining the shard identification information corresponding to each of the second data shards includes: Determine each sub-slice contained in each second data slice, and determine the first byte offset of each sub-slice corresponding to the target data and the length of each sub-slice, wherein the data in the same sub-slice comes from the untransmitted data corresponding to the same target first data slice; For any of the second data shards, generate shard identification information corresponding to the second data shard, the shard identification information at least including the first byte offset corresponding to each of the sub-shards, the length of each of the sub-shards, and the identification of the target first data shard corresponding to each of the sub-shards.

7. The method according to claim 6, characterized in that The respectively determining each sub-shard contained in each second data shard includes: respectively determining a second byte offset of each of the untransmitted data in the target untransmitted data, and respectively determining a third byte offset of each of the second data fragments in the target untransmitted data; According to the second byte offset and the third byte offset, each sub-shard contained in each second data shard is determined respectively.

8. The method according to claim 1, characterized in that The method further comprises: Displaying transmission progress information on a display interface, wherein the transmission progress information is used to indicate the size of data sent to the data receiving end; Upon receiving transmission completion information sent by the data receiving end, updating the transmission progress information according to the transmission completion information; the transmission completion information is sent by the data receiving end when the transmission of any of the first data fragments is completed; When the transmission suspension event is triggered, the transmission progress information is updated according to the untransmitted data corresponding to each of the target first data segments.

9. A data transmission device, characterized in that: The device comprises: A first transmission module, configured to transmit, in parallel, first data fragments of target data to a data receiving end through multiple threads; a determination module, configured to, when a transmission suspension event is triggered, terminate the transmission of each of the first data slices, determine target first data slices for which transmission has not been completed in each of the first data slices, and untransmitted data corresponding to each of the target first data slices; a sharding module configured to, upon triggering a transmission restart event, re-shard the mixed data to obtain second data shards, and respectively determine shard identification information corresponding to each second data shard; wherein the shard identification information includes at least a target first data shard identification and a splicing order identification of the target first data shard identification, the target first data shard identification being an identification of the target first data shard to which the data in the second data shard belongs, and the splicing order indicating the second data shards to which the untransmitted data in the target first data shard belongs, as well as the corresponding splicing order of the data in the second data shards corresponding to the target first data shard when the target first data shard is spliced; The second transmission module is configured to transmit the second data slices and the slice identification information of the second data slices to the data receiving end in parallel through multiple threads.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images