Method and device for processing failed data in the middle of large batches of data

Abstract

The present application provides a method and device for processing failed data in transit for large batches of data, and relates to the field of communication technology. The method includes: the sending end receives the data receiving response sent by the receiving end according to a preset period while continuously sending data packets to the receiving end; based on the sending time of the successfully transmitted business data corresponding to each data packet, the data packet that failed to be transmitted and the failed data that failed to be transmitted in the corresponding data packet are determined; each failed data is resent to the receiving end through one or more configured retransmission channels according to the sending time sequence of each failed data and the breakpoint retransmission mechanism. The method can retransmit the failed data that failed to be transmitted in a targeted manner, truly avoiding unnecessary retransmissions, further avoiding the waste of resources in the data retransmission process, and improving data transmission efficiency.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a method and device for processing failed data in transit for large batches of data. Background Art

[0002] For the massive amount of information collected from social media, news websites and other platforms, during the transmission of massive data and the subsequent management process, data can be transmitted based on the real-time transport protocol at the sending and receiving ends; however, during the data transmission process, data transmission may fail due to a failure at the sending end or a network failure. Most of the retransmission schemes in the prior art are based on timed and quantitative retransmission strategies such as timers, packet loss rates, and retransmission data thresholds. The core of these schemes is to control the retransmission data according to the set threshold. When the threshold is reached, the sending of the retransmission data is slowed down or prohibited.

[0003] The above-mentioned traditional data retransmission scheme, on the one hand, does not go into the selection of retransmission messages, and transmits all retransmission data without paying attention to the problem of resource waste; on the other hand, it only controls the sending of retransmission data units in a timely and quantitative manner, and does not really avoid unnecessary retransmissions. There is still a problem of resource waste during retransmission. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method and device for processing failed data in mid-transmission for large quantities of data, so as to solve the above-mentioned problems existing in the prior art and avoid waste of resources during data retransmission.

[0005] In a first aspect, a method for processing failed data in midstream for a large amount of data is provided, and the method may include:

[0006] The transmitting end receives a data receiving response sent by the receiving end according to a preset period while continuously sending a data packet to the receiving end; the data packet includes a plurality of service data; the data receiving response includes the sending time of the successfully transmitted service data;

[0007] Determine the data packets that failed to be transmitted and the failure data that failed to be transmitted in the corresponding data packets based on the sending time of the successfully transmitted business data corresponding to each data packet;

[0008] Each failed data is resent to the receiving end through one or more configured retransmission channels according to the sending time sequence of each failed data and the breakpoint retransmission mechanism.

[0009] In a possible implementation, determining the number of the retransmission channels includes:

[0010] For any data packet that fails to be transmitted, predict the retransmission duration of each failed data in the data packet;

[0011] Determine the data packet retransmission duration of the data packet based on the retransmission duration of each failed data;

[0012] The number of retransmission channels is determined based on the retransmission duration of each data packet and the number of data packets.

[0013] In a possible implementation, determining the number of retransmission channels based on the retransmission duration of each data packet and the number of data packets includes:

[0014] If the number of packets is 1, the number of retransmission channels is determined to be 1;

[0015] If the number of packets is greater than 1, the retransmission duration of the data packets that meet the combination condition is determined, and the corresponding multiple data packets are combined to determine a combined data packet; based on the combined data packet, the number of retransmission channels is determined.

[0016] In a possible implementation, after determining the data packet that fails to be transmitted and the failure data in the corresponding data packet that fails to be transmitted, the method further includes:

[0017] When the amount of data packets sent by the sending end in the historical period corresponding to the preset period is unqualified, the amount of failed data in the preset period is reduced by adjusting the sending time interval of the remaining data packets among all the data packets sent by the sending end.

[0018] In a possible implementation, determining that the amount of data packets sent by the sending end in a historical period corresponding to the preset period is unqualified includes:

[0019] Obtaining the number of failed data within the preset period;

[0020] If the data quantity is not greater than the first preset quantity, determining that the amount of data packets sent by the sending end in the historical period corresponding to the preset period is qualified;

[0021] If the data quantity is greater than the first preset quantity and not greater than the second preset quantity, determining that the amount of data packets sent by the sending end within the historical period is in a pending confirmation state;

[0022] If the data quantity is greater than a second preset data quantity, it is determined that the sending amount of data packets sent by the sending end within the historical period is unqualified.

[0023] In a possible implementation, the data reception response further includes a reception time of each service data received by the receiving end;

[0024] When the amount of data packets sent by the sending end in the historical period is in a pending confirmation state, the method further includes:

[0025] Determine the average transmission rate of the successfully transmitted service data based on the reception time, transmission time and data size of each service data;

[0026] If the average transmission rate is lower than the standard transmission rate, adjusting the time interval for sending the remaining data packets among all the data packets sent by the sending end;

[0027] If the average transmission rate is not less than the standard transmission rate, it is determined that the amount of data packets sent by the sending end during the historical period is unqualified.

[0028] In a possible implementation, adjusting the sending time interval of the remaining data packets in all the data packets includes:

[0029] Calculating the average transmission rate and the standard transmission rate to obtain a target rate difference;

[0030] Determine the target adjustment coefficient corresponding to the target rate difference based on the corresponding relationship between the configured rate difference range and the adjustment coefficient;

[0031] The target adjustment coefficient is used to adjust the sending time interval to obtain the target interval, so as to control the sending end to send the remaining data packets in all data packets according to the target interval.

[0032] In a second aspect, a device for processing failed data in midstream for large batches of data is provided, and the device may include:

[0033] A receiving unit, configured to receive a data receiving response sent by the receiving end according to a preset period during the process of the sending end continuously sending a data packet to the receiving end; the data packet includes a plurality of service data; the data receiving response includes the sending time of the successfully transmitted service data;

[0034] A determination unit, configured to determine the data packets that failed to be transmitted and the failure data that failed to be transmitted in the corresponding data packets based on the sending time of the successfully transmitted service data corresponding to each data packet;

[0035] The sending unit is used to resend each failed data to the receiving end through one or more configured retransmission channels according to the sending time sequence of each failed data and the breakpoint retransmission mechanism.

[0036] In a third aspect, an electronic device is provided, the electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0037] Memory, used to store computer programs;

[0038] The processor is used to implement any method step described in the first aspect when executing the program stored in the memory.

[0039] In a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any method step described in the first aspect is implemented.

[0040] The present application provides a method for processing failed data in transit for large batches of data, the method comprising: the sending end receives the data receiving response sent by the receiving end according to a preset period while continuously sending data packets to the receiving end; based on the sending time of the successfully transmitted business data corresponding to each data packet, the data packet that failed to be transmitted and the failed data that failed to be transmitted in the corresponding data packet are determined; each failed data is resent to the receiving end through one or more configured retransmission channels according to the sending time sequence of each failed data and the breakpoint retransmission mechanism. This method can retransmit the business data (failed data) that failed to be transmitted in a targeted manner, truly avoiding unnecessary retransmissions, further avoiding the waste of resources in the data retransmission process, and improving data transmission efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A system architecture diagram for a method for processing failed data in transit for large batches of data provided in an embodiment of the present application;

[0043] Figure 2 A flowchart of a method for processing failed data in bulk provided by an embodiment of the present application;

[0044] Figure 3 A schematic diagram of the structure of a device for processing failed data in the middle of a large batch of data provided by an embodiment of the present application;

[0045] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] The method for processing failed data in the middle of a large batch of data provided by the embodiment of the present application can be applied to Figure 1 In the system architecture shown in Figure 1 As shown, it may include: a sending end and a receiving end.

[0048] The sending end is used to divide a large batch of business data into multiple data packets, send the multiple data packets to the receiving end at preset time intervals, and mark the sending time of multiple business data in each data packet.

[0049] The receiving end is used to send a data receiving response to the sending end, wherein the data receiving response includes the sending time of the successfully transmitted service data.

[0050] The sending end is also used to receive a data receiving response sent by the receiving end according to a preset period, so as to execute a method for processing midway failed data for large batches of data provided by the present application.

[0051] In the field of communication technology, the sender and the receiver can transmit data based on the real-time transport protocol; however, during the data transmission process, data transmission may fail due to a failure of the sender or a network failure. Most of the retransmission schemes in the prior art are based on timed and quantitative retransmission strategies such as timers, packet loss rates, and retransmission data thresholds. The core of the retransmission scheme is to control the retransmission data according to the set threshold. When the threshold is reached, the sending of the retransmission data is slowed down or prohibited.

[0052] The above-mentioned traditional data retransmission scheme, on the one hand, does not go into the selection of retransmission messages, and transmits all retransmission data without paying attention to the problem of resource waste; on the other hand, it only controls the sending of retransmission data units in a timely and quantitative manner, and does not really avoid unnecessary retransmissions. There is still a problem of resource waste during retransmission.

[0053] Therefore, the present application extracts a method for processing failed data in mid-transmission for large batches of data, which is used to solve the above-mentioned problems existing in the prior art and avoid waste of resources during data retransmission.

[0054] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0055] Figure 2 A flowchart of a method for processing failed data in bulk data provided by an embodiment of the present application. Figure 2 As shown, the method may include:

[0056] Step S210: While continuously sending data packets to the receiving end, the sending end receives a data receiving response sent by the receiving end according to a preset period.

[0057] The data packet includes a plurality of business data; the data receiving response includes the sending time of the successfully transmitted business data.

[0058] Specifically, the sending end divides a large batch of business data into multiple data packets, which can be divided according to the data type of the business data or the data size of the business data.

[0059] Afterwards, the sender sends multiple data packets to the receiver according to the configured sending time interval.

[0060] The receiving end generates a data receiving response according to the data identifier of the successfully transmitted business data, and sends the data receiving response to the sending end; it can be understood that the data identifier is the sending time of the business data sent by the sending end.

[0061] The sending end receives the data receiving response sent by the receiving end according to a preset period.

[0062] Step S220: Determine the data packets that failed to be transmitted and the failure data that failed to be transmitted in the corresponding data packets based on the sending time of the successfully transmitted business data corresponding to each data packet.

[0063] Specifically, the sending end retains the service data in all data packets;

[0064] The sending time of the successfully transmitted service data received by the sending end from the receiving end;

[0065] Since the sender marks the sending time of each business data and the sending time and end time of each data packet (that is, the sending time of the first business data and the sending time of the last business data in the corresponding data packet) when sending each business data, the business data that failed to be transmitted, that is, the failed data, is determined according to the sending time of the successfully transmitted business data fed back by the receiving end; if all the business data in a data packet fails to be transmitted, the data packet that failed to be transmitted is determined; if part of the business data in a data packet fails to be transmitted, the failed data in the corresponding data packet is determined.

[0066] Step S230: resend each failed data to the receiving end through one or more configured retransmission channels according to the sending time sequence of each failed data and the breakpoint retransmission mechanism.

[0067] Specifically, the sending time of the failed data in the corresponding data packet is obtained; the failed data in each data packet is sorted according to the sending time to obtain a data packet including a plurality of failed data arranged in an orderly manner;

[0068] It should be noted that the data packet that fails to be transmitted may be a data packet corresponding to a data packet in which all business data in the initially sent data packet fails to be transmitted, or may be a data packet in which part of the business data in the initially sent data packet fails to be transmitted.

[0069] Among them, the breakpoint retransmission mechanism allows the breakpoint resuming function when processing files (data), which is particularly important when using Spooling Directory Source to monitor and read the sender's files. This mechanism ensures that even if the file transfer fails or restarts, the file can be read from the last stop position, avoiding data duplication or loss.

[0070] The key to implementing breakpoint resumable transfer: Position File, which implements breakpoint resumable transfer by maintaining an offset (position file). This offset records the current position of each file read during file transfer;

[0071] Furthermore, the file offset (pos) indicates the read position in the file. Each time a file transfers a portion of data, it updates this offset to reflect where the next transfer should start. This offset records the transferred files, each of which contains the absolute directory and inode file unique identifier. pos is the core representation of the byte offset.

[0072] That is to say, the failed data uses the breakpoint retransmission mechanism during the retransmission process.

[0073] In some embodiments, the specific process of determining the number of retransmission channels may include:

[0074] For any data packet that fails to transmit, the retransmission duration of each failed data in the data packet is predicted; further, the retransmission duration of each failed data can be predicted based on the current network transmission rate and the data size of the corresponding failed data.

[0075] In some embodiments, the retransmission duration of each failed data is predicted by a duration prediction model; the training process of the duration prediction model can be: extracting meaningful features from historical failure data, such as data size (data size determined by the breakpoint retransmission mechanism, not the original data size), source IP address, destination IP address, protocol type, etc. as training samples; using the training samples to train the models corresponding to Random Forest Regression (Random Forest Regression), Gradient Boosting Regression Trees (GradientBoosting Regression Trees, GBRT), Support Vector Regression (SVR) and Neural Network, respectively, to perform hyperparameter tuning for each model to obtain the best performance, and use k-fold cross validation to evaluate the performance of each model; assign weights to each model, and perform weighted averaging of their prediction results to obtain the final prediction. "Vote" for each model, majority decision or weighted voting; mean squared error (MSE), root mean squared error (RMSE) and determination coefficient (R² score) indicators can also be used to evaluate model performance, that is, an integrated duration prediction model is obtained.

[0076] Furthermore, the process of assigning weights to each model may include: Assuming there are three models A, B, and C, and their performances on the validation set are MSE_A, MSE_B, and MSE_C, respectively. First, the inverse of each model is calculated as the initial weight, and then normalized:

[0077] Initial weights:

[0078] Normalized weights:

[0079] Where n is the number of models; the weight of each model is determined according to the normalized weight.

[0080] Based on the retransmission duration of each failed data, the data packet retransmission duration of the data packet is determined; the data packet retransmission duration is determined based on the accumulation of the retransmission durations of each failed data in the corresponding data packet.

[0081] The number of retransmission channels is determined based on the retransmission duration of each data packet and the number of packets of the data packet. The process may specifically include: if the number of packets is 1, the number of retransmission channels is determined to be 1.

[0082] If the number of packets is greater than 1, it means that there are multiple data packets that have failed to be transmitted and are waiting to be transmitted again. At this time, the retransmission duration of the data packets that meet the combination conditions is determined, and the corresponding multiple data packets are combined to determine the combined data packets; based on the combined data packets, the number of retransmission channels is determined. For example: the data packet retransmission duration TA of data packet A is 30s, the data packet retransmission duration TB of data packet B is 60s, and the data packet retransmission duration TC of data packet C is 20s. At this time, data packets A and data packet C are combined, that is, TA+TC<TB; the combination process does not combine the business data therein, but places data packets A and data packet C on the same retransmission channel for transmission in sequence, and data packet B is placed on a retransmission channel for transmission, thereby determining the number of retransmission channels to be 2. Generally, the maximum number of retransmission channels is 3.

[0083] In some embodiments, when the number of packets is 1 and the data packet retransmission time of the data packet is greater than the transmission time of the remaining data packets in all data packets, the business data in the data packet that failed to transmit is grouped to determine multiple groups of business data that need to be retransmitted; according to the number of groups, the number of retransmission channels is determined. For example; there are 10 data packets in total, 7 data packets have been sent to the receiving end, and 3 data packets are left to be transmitted; the current retransmitted data packet is 1, of which there are 3 business data packets that need to be retransmitted; the data packet retransmission time of the retransmitted data packet is 60s, but the transmission time of the remaining 3 data packets is about 40s, which indicates that the data packet retransmission time is greater than the transmission time of the remaining data packets in all data packets. The 3 business data that need to be retransmitted are grouped, and the grouping process can be grouped based on the predicted retransmission time of these 3 business data (the retransmission time of business data 1 is 15s, the retransmission time of business data 2 is 10s, and the retransmission time of business data 3 is 35s), and the grouping result is obtained (business data 1 and business data 2 are one group; business data 3 is another group), and the two groups of business data are transmitted through 2 retransmission channels, and the data packet retransmission time of the retransmitted data packet is controlled to 35s, so that when the transmission of the remaining 3 data packets is completed, the transmission of the data packet that needs to be retransmitted is also completed.

[0084] In some embodiments, in order to reduce the amount of failed data, the cause of the failed business data is confirmed, and the specific process may include:

[0085] The data receiving response also includes the receiving time of each service data received by the receiving end, and the sending end is also configured with a first preset amount and a second preset amount of data, wherein the first preset amount is less than the second preset amount of data;

[0086] The sender obtains the number of failed data within a preset period;

[0087] A. If the amount of data is not greater than the first preset amount, it is determined that the amount of data packets sent by the sender within the historical period corresponding to the preset period is qualified; it can be understood that the historical period is the period when multiple business data received by the receiving end within the preset period are sent by the sender.

[0088] B. If the data quantity is greater than the first preset quantity and not greater than the second preset quantity, it is determined that the amount of data packets sent by the sender within the historical period is in a pending confirmation state.

[0089] Furthermore, the sending end is also configured with a standard transmission rate, which is a rate at which business data can be stably transmitted as determined by professionals based on historical data.

[0090] First, based on the reception time, transmission time and data size of each business data, the transmission rate of each successfully transmitted business data is determined, thereby determining the average transmission rate of multiple successfully transmitted business data;

[0091] Afterwards, if the average transmission rate is lower than the standard transmission rate, the sending end adjusts the time interval for sending the remaining data packets among all the data packets; the adjustment method generally sets the adjustment coefficient to 1.15, that is, increases the sending end's sending time interval for sending the remaining data packets.

[0092] In some embodiments, the adjustment method may also be:

[0093] Calculate the average transmission rate and the standard transmission rate to obtain the target rate difference;

[0094] Based on the correspondence between the configured rate difference range and the adjustment coefficient, the target adjustment coefficient corresponding to the target rate difference is determined; the larger the target rate difference, the larger the target adjustment coefficient. This correspondence is generally determined by professionals based on historical data. Since the standard transmission rates in different scenarios are different, the rate difference range settings are also different. Similarly, the corresponding adjustment coefficient will change with the standard transmission rate; the adjustment coefficient is greater than 1.

[0095] The target adjustment coefficient α is used to adjust the sending time interval T to obtain the target interval T', so as to control the sending end to send the remaining data packets according to the target interval; the adjustment process is T'=T×α. After the adjustment is completed, return to step S210.

[0096] If the average transmission rate is not less than the standard transmission rate, it is determined that the amount of data packets sent by the sending end within the preset period is unqualified.

[0097] C. If the data quantity is greater than the second preset data quantity, it is determined that the number of data packets sent by the sending end within the historical period is unqualified.

[0098] When the amount of data packets sent by the sending end in the historical period is unqualified, the method may further include:

[0099] Calculating the transmission amount and the second preset data amount to determine an absolute value of a difference between the transmission amount and the second preset data amount;

[0100] Based on the ratio β of the absolute value of the difference and the preset difference, the sending time interval T is adjusted to obtain the target time interval T'', T''=T×β, and the sending amount of data packets sent in the preset period is reduced by increasing the sending time interval, and the execution returns to step S210. If the target interval reaches the set maximum sending time interval, and the number of data packets sent by the sender in the historical period is still unqualified, an alarm indication is issued to the corresponding processing person.

[0101] The present application provides a method for processing failed data in transit for large batches of data, the method comprising: the sending end receives the data receiving response sent by the receiving end according to a preset period while continuously sending data packets to the receiving end; based on the sending time of the successfully transmitted business data corresponding to each data packet, the data packet that failed to be transmitted and the failed data that failed to be transmitted in the corresponding data packet are determined; each failed data is resent to the receiving end through one or more configured retransmission channels according to the sending time sequence of each failed data and the breakpoint retransmission mechanism. This method can retransmit the business data (failed data) that failed to be transmitted in a targeted manner, truly avoiding unnecessary retransmissions, further avoiding the waste of resources in the data retransmission process, and improving data transmission efficiency.

[0102] Corresponding to the above method, the embodiment of the present application also provides a device for processing failed data in the middle of a large batch of data, such as Figure 3 As shown, the device comprises:

[0103] The receiving unit 310 is used to receive a data receiving response sent by the receiving end according to a preset period during the process of the sending end continuously sending a data packet to the receiving end; the data packet includes a plurality of service data; the data receiving response includes the sending time of the successfully transmitted service data;

[0104] A determination unit 320, configured to determine the data packets that failed to be transmitted and the failure data that failed to be transmitted in the corresponding data packets based on the sending time of the successfully transmitted service data corresponding to each data packet;

[0105] The sending unit 330 is used to resend each failed data to the receiving end through one or more configured retransmission channels according to the sending time sequence of each failed data and the breakpoint retransmission mechanism.

[0106] The functions of each functional unit of a data processing device for failed midway data provided in the above-mentioned embodiment of the present application can be realized through the above-mentioned method steps. Therefore, the specific working process and beneficial effects of each unit in a data processing device for failed midway data provided in the embodiment of the present application are not repeated here.

[0107] The present application also provides an electronic device, such as Figure 4 As shown, it includes a processor 410 , a communication interface 420 , a memory 430 and a communication bus 440 , wherein the processor 410 , the communication interface 420 , and the memory 430 communicate with each other via the communication bus 440 .

[0108] Memory 430, for storing computer programs;

[0109] The processor 410 is used to execute the program stored in the memory 430 to implement the following steps:

[0110] The transmitting end receives a data receiving response sent by the receiving end according to a preset period while continuously sending a data packet to the receiving end; the data packet includes a plurality of service data; the data receiving response includes the sending time of the successfully transmitted service data;

[0111] Determine the data packets that failed to be transmitted and the failure data that failed to be transmitted in the corresponding data packets based on the sending time of the successfully transmitted business data corresponding to each data packet;

[0112] Each failed data is resent to the receiving end through one or more configured retransmission channels according to the sending time sequence of each failed data and the breakpoint retransmission mechanism.

[0113] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0114] The communication interface is used for communication between the above electronic device and other devices.

[0115] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0116] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0117] The implementation methods and beneficial effects of the components of the electronic device in the above embodiments to solve the problems can be seen in Figure 2 The various steps in the illustrated embodiment are implemented, therefore, the specific working process and beneficial effects of the electronic device provided by the embodiment of the present application are not repeated here.

[0118] In another embodiment provided in the present application, a computer-readable storage medium is also provided, which stores instructions. When the computer-readable storage medium is executed on a computer, the computer executes a method for processing failed data for large batches of data as described in any of the above embodiments.

[0119] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute a method for processing failed data in midstream for large batches of data as described in any of the above embodiments.

[0120] Those skilled in the art will appreciate that the embodiments in the present application can be provided as methods, systems, or computer program products. Therefore, the embodiments in the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments in the present application can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0121] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0122] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0123] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0124] Unless otherwise defined, the technical terms or scientific terms used in this application should be understood by people with ordinary skills in the field to which the present invention belongs. "First", "second" and similar words used in this application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect", "couple" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0125] Although preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they understand the basic creative concepts.

[0126] Obviously, those skilled in the art can make various changes and modifications to the embodiments in the present application without departing from the spirit and scope of the embodiments in the present application. Thus, if these modifications and variations of the embodiments in the present application are within the scope of the embodiments in the present application and their equivalents, the embodiments in the present application are also intended to include these modifications and variations.

Claims

1. A method for processing failed data in the middle of a large batch of data, characterized in that: The method comprises: The transmitting end receives a data receiving response sent by the receiving end according to a preset period while continuously sending a data packet to the receiving end; the data packet includes a plurality of service data; the data receiving response includes the sending time of the successfully transmitted service data; Determine the data packets that failed to be transmitted and the failure data that failed to be transmitted in the corresponding data packets based on the sending time of the successfully transmitted business data corresponding to each data packet; Resending each failed data to the receiving end through one or more configured retransmission channels according to the sending time sequence of each failed data and the breakpoint retransmission mechanism; After determining the data packet that failed to be transmitted and the failed data in the corresponding data packet that failed to be transmitted, the method further includes: When the amount of data packets sent by the sending end in the historical period corresponding to the preset period is unqualified, the amount of failed data in the preset period is reduced by adjusting the time interval for sending the remaining data packets among all the data packets sent by the sending end; Determining that the amount of data packets sent by the sending end in the historical period corresponding to the preset period is unqualified includes: Obtaining the number of failed data within the preset period; If the data quantity is not greater than the first preset quantity, determining that the amount of data packets sent by the sending end in the historical period corresponding to the preset period is qualified; If the data quantity is greater than the first preset quantity and not greater than the second preset quantity, determining that the amount of data packets sent by the sending end within the historical period is in a pending confirmation state; If the data quantity is greater than a second preset data quantity, determining that the amount of data packets sent by the sending end during the historical period is unqualified; The data receiving response also includes the receiving time of each service data received by the receiving end; When the amount of data packets sent by the sending end in the historical period is in a pending confirmation state, the method further includes: Determine the average transmission rate of the successfully transmitted service data based on the reception time, transmission time and data size of each service data; If the average transmission rate is lower than the standard transmission rate, adjusting the time interval for sending the remaining data packets among all the data packets sent by the sending end; If the average transmission rate is not less than the standard transmission rate, it is determined that the amount of data packets sent by the sending end during the historical period is unqualified.

2. The method according to claim 1, characterized in that Determining the number of the retransmission channels includes: For any data packet that fails to be transmitted, predict the retransmission duration of each failed data in the data packet; Determine the data packet retransmission duration of the data packet based on the retransmission duration of each failed data; The number of retransmission channels is determined based on the retransmission duration of each data packet and the number of data packets.

3. The method according to claim 2, characterized in that Determining the number of retransmission channels based on the retransmission duration of each data packet and the number of data packets includes: If the number of packets is 1, the number of retransmission channels is determined to be 1; If the number of packets is greater than 1, the retransmission duration of the data packets that meet the combination condition is determined, and the corresponding multiple data packets are combined to determine a combined data packet; based on the combined data packet, the number of retransmission channels is determined.

4. The method according to claim 1, characterized in that Adjusting the sending time interval of the remaining data packets in all data packets includes: Calculating the average transmission rate and the standard transmission rate to obtain a target rate difference; Determine the target adjustment coefficient corresponding to the target rate difference based on the corresponding relationship between the configured rate difference range and the adjustment coefficient; The target adjustment coefficient is used to adjust the sending time interval to obtain the target interval, so as to control the sending end to send the remaining data packets in all data packets according to the target interval.

5. A data processing device for failed data in the middle of a large batch of data, characterized in that: The device comprises: A receiving unit, configured to receive a data receiving response sent by the receiving end according to a preset period during the process of the sending end continuously sending a data packet to the receiving end; the data packet includes a plurality of service data; the data receiving response includes the sending time of the successfully transmitted service data; A determination unit, configured to determine the data packets that failed to be transmitted and the failure data that failed to be transmitted in the corresponding data packets based on the sending time of the successfully transmitted service data corresponding to each data packet; A sending unit, configured to resend each failed data to the receiving end through one or more configured retransmission channels according to the sending time sequence of each failed data and the breakpoint retransmission mechanism; After determining the data packet that failed to be transmitted and the failed data in the corresponding data packet that failed to be transmitted, the device further includes an adjustment unit. The adjusting unit is used to reduce the amount of failed data in the preset period by adjusting the sending time interval of the remaining data packets among all the data packets sent by the sending end when the sending amount of data packets sent by the sending end in the historical period corresponding to the preset period is unqualified; Determining that the amount of data packets sent by the sending end in the historical period corresponding to the preset period is unqualified includes: Obtaining the number of failed data within the preset period; If the data quantity is not greater than the first preset quantity, determining that the amount of data packets sent by the sending end in the historical period corresponding to the preset period is qualified; If the data quantity is greater than the first preset quantity and not greater than the second preset quantity, determining that the amount of data packets sent by the sending end within the historical period is in a pending confirmation state; If the data quantity is greater than a second preset data quantity, determining that the amount of data packets sent by the sending end during the historical period is unqualified; The data receiving response also includes the receiving time of each service data received by the receiving end; When the amount of data packets sent by the transmitting end in the historical period is in a pending confirmation state, the determining unit is further used to determine the average transmission rate of the successfully transmitted service data according to the receiving time, sending time and data size of each service data; If the average transmission rate is lower than the standard transmission rate, adjusting the time interval for sending the remaining data packets among all the data packets sent by the sending end; If the average transmission rate is not less than the standard transmission rate, it is determined that the amount of data packets sent by the sending end during the historical period is unqualified.

Images