Data compensation method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a data compensation method and device, electronic equipment and storage medium, relates to the technical field of computers, in particular to the technical field of data processing and data transmission. The specific implementation scheme is as follows: determining a current compensation failure rate of current to-be-compensated data, the compensation failure rate being used to represent the proportion of data processing failure relative to the total data processing; in the case that the compensation failure rate is less than a preset threshold, processing the to-be-compensated data in real time; in the case that the compensation failure rate is not less than the preset threshold, processing the to-be-compensated data with a time delay. The generation of the loop compensation loop failure problem can be reduced.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to the field of data processing and data transmission technology. Background Technology

[0002] In daily business operations, in some cases, no additional processing is required after data consumption fails; however, in other cases, data compensation is needed for the failed data, that is, the failed data is consumed again. Summary of the Invention

[0003] This disclosure provides a data compensation method, apparatus, electronic device, and storage medium.

[0004] According to a first aspect of this disclosure, a data compensation method is provided, comprising:

[0005] Determine the current compensation failure rate of the current data to be compensated, whereby the compensation failure rate represents the proportion of data processing failures relative to the total amount of data processing.

[0006] If the compensation failure rate is less than a preset threshold, the data to be compensated is processed in real time.

[0007] If the compensation failure rate is not less than a preset threshold, the data to be compensated is delayed.

[0008] According to a second aspect of this disclosure, a data compensation apparatus is provided, comprising:

[0009] The determination module is used to determine the current compensation failure rate of the current data to be compensated, wherein the compensation failure rate is used to represent the proportion of data processing failures relative to the total data processing.

[0010] The real-time processing module is used to process the data to be compensated in real time when the compensation failure rate is less than a preset threshold.

[0011] The delay processing module is used to perform delay processing on the data to be compensated when the compensation failure rate is not less than a preset threshold.

[0012] According to a third aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and

[0013] A memory communicatively connected to the at least one processor; wherein,

[0014] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described in any of the first aspects.

[0015] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform any of the steps of the method described in the first aspect.

[0016] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of any of the methods described in the first aspect.

[0017] This disclosure provides a data compensation method, apparatus, electronic device, and storage medium that can reduce the occurrence of loop compensation loop failures.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0020] Figure 1 This is a flowchart of data compensation in the related technology provided in the embodiments of this disclosure;

[0021] Figure 2 This is a flowchart of a data compensation method provided in an embodiment of this disclosure;

[0022] Figure 3 This is another flowchart of the data compensation method provided in this embodiment of the disclosure;

[0023] Figure 4 This is another flowchart of the data compensation method provided in the embodiments of this disclosure;

[0024] Figure 5 This is a flowchart of a data compensation method provided in an embodiment of this disclosure;

[0025] Figure 6 This is a schematic diagram of a data compensation device provided in an embodiment of this disclosure;

[0026] Figure 7 This is a block diagram of an electronic device used to implement the data compensation method of the embodiments of this disclosure. Detailed Implementation

[0027] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0028] Data consumption can be understood as data processing, which involves processing data according to the request information it carries. For example, when searching for "mobile phone" on a shopping website on an electronic device, the server receives the request data containing "mobile phone" and sends all relevant product information about mobile phones to the device, thus completing the processing of the requested data. Data processing can result in success or failure. Data compensation can be understood as reprocessing data in the event of processing failure. For example, failed data can be stored in a compensation queue as data to be compensated, and then retrieved from the compensation queue for reprocessing.

[0029] In daily business operations, there are two scenarios. In one, some business processes don't have strict requirements regarding data processing failures; a retry is sufficient. For example, when requesting to view a news list, if data processing fails, the user can retry by clicking the retry button to resend the data. In another scenario, if data processing fails, corresponding compensation measures are needed, i.e., data compensation. For example, in a ranking activity, users earn points for completing a task, such as watching a video for a certain duration, which automatically awards points. This point-awarding action might fail, and the user is unaware of it—it can be understood as there being no retry button. In this case, data compensation is needed for the failed data, i.e., reprocessing.

[0030] In related technologies, data compensation typically employs methods such as... Figure 1 As shown, the data to be compensated is typically stored in a compensation queue, as indicated. Figure 1 The data compensation process is explained below.

[0031] Step 1: Start the data compensation task;

[0032] Step 2: Determine if there is any data to be compensated;

[0033] Step 3: If not, end the data compensation task;

[0034] Step 4: If so, repeatedly retrieve the data to be compensated;

[0035] The system can periodically retrieve data to be compensated from the compensation queue and process it in a loop. For example, at preset intervals, it can check the data to be compensated in the compensation queue and retrieve the data to be compensated in the compensation queue in sequence until all the data to be compensated is retrieved, that is, there is no data to be compensated in the compensation queue, then the data compensation task ends.

[0036] Step 5, data processing;

[0037] After obtaining the data to be compensated, data processing is performed on the data to be compensated, that is, data compensation is carried out.

[0038] Step 6: Determine if the processing was successful;

[0039] Step 7: If yes, decrease the amount to be compensated by 1;

[0040] Step 8: If not, increment the data to be compensated by 1;

[0041] If the data to be compensated fails to be processed, the data to be compensated will be added back to the compensation queue, i.e., the data to be compensated in the compensation queue will be incremented by 1; if the processing is successful, the data to be compensated in the compensation queue will be decremented by 1.

[0042] Step 9: Process the request data for normal business operations;

[0043] Normal business requests consist of data other than the data to be compensated; this can be understood as the data processed during the first request.

[0044] Return to step 6 to determine if the processing was successful; if not, return to step 8.

[0045] This step is used to determine whether the request data for normal business operations has been processed successfully.

[0046] If a normal business request fails to process, the data is added to the compensation queue as data to be compensated.

[0047] Perform the above steps for each piece of data to be compensated retrieved until the data compensation task is completed.

[0048] Reference Figure 1In related technologies, the process of compensating (processing) data is continuous. It can be understood that once a piece of data to be compensated is processed, regardless of success or failure, if there is still data to be compensated stored in the compensation queue, it needs to continue fetching and processing this data—this is called cyclical compensation. If the number of requests for processing normal business data and data to be compensated within one second exceeds the service interface's Queries Per Second (QPS), the cyclical compensation will fail. QPS can be understood as the number of data requests a service interface can handle per second, including both normal business request data and data to be compensated. As an example, if sending one piece of data to the service interface every 500 milliseconds (ms) allows for normal processing, meaning the processing time for a successfully processed piece of data is 500ms, and the service interface's QPS is 2 (meaning only a maximum of two pieces of data can be processed normally per second, and any data exceeding this limit will fail), then sending a request every 500ms usually works. However, if processing fails due to the number of data exceeding the service interface's QPS, the failed data needs to be placed in a compensation queue to prevent data loss. But if three pieces of data are sent simultaneously within one second, one piece of data will fail to process, and this failed data will also be placed in the compensation queue to await compensation. When the number of data awaiting compensation and the number of normal business requests simultaneously exceed the service interface's QPS within one second, a loop compensation loop failure problem will occur.

[0049] Another scenario is that the QPS of the service interface does not have an explicit limit, but may be limited based on a certain business attribute. When the data sent to the service interface for processing exceeds this limit, it may also cause the loop compensation loop to fail. For example, the limit may be based on the user identifier (uid). The amount of data related to the same uid cannot exceed a preset number, such as 2 or 3. If the preset number is exceeded, the loop compensation loop will fail. The data related to the same uid can include normal business request data and data to be compensated related to that uid.

[0050] To address the issue of loop failure during data compensation, this disclosure provides a data compensation method, apparatus, electronic device, and storage medium that can be applied to any business involving data compensation.

[0051] Figure 2 A flowchart of a data compensation method provided in an embodiment of this disclosure, such as... Figure 2 As shown, it includes:

[0052] Step S201: Determine the current compensation failure rate of the current data to be compensated. The compensation failure rate is used to represent the proportion of data processing failures relative to the total data processing.

[0053] Step S202: If the compensation failure rate is less than a preset threshold, process the data to be compensated in real time.

[0054] Step S203: If the compensation failure rate is not less than a preset threshold, the data to be compensated is delayed.

[0055] In this embodiment, the current compensation failure rate of the data to be compensated is determined. If the compensation failure rate is less than a preset threshold, the data to be compensated is processed in real time; if the compensation failure rate is not less than the preset threshold, the data to be compensated is processed with a delay. In related technologies, data to be compensated is directly processed after being retrieved from the compensation queue. When the total number of requests sent to the service interface for processing simultaneously exceeds the QPS of the service interface, the processing of the data to be compensated will fail. The failed data to be compensated will re-enter the compensation queue as new data to be compensated, and after another processing failure, it will be retrieved from the compensation queue again for processing, which may fail again. Therefore, there may be a situation of cyclical compensation failure. Compared with the prior art, this embodiment takes different measures depending on whether the compensation failure rate of the data to be compensated is less than the preset threshold. When the compensation failure rate is not less than the preset threshold, the data to be compensated is processed with a delay, which can alleviate the situation where the total number of requests sent to the service interface for processing simultaneously exceeds the QPS of the service interface, thus reducing the occurrence of cyclical compensation failure.

[0056] Furthermore, since the data compensation methods in related technologies can lead to the problem of loop compensation failure, that is, the data to be compensated fails to be processed and then re-enters the compensation queue to wait for processing, resulting in low compensation efficiency and the occupation of more compensation queue storage resources, the embodiments of this disclosure can reduce the occurrence of loop compensation failure, thereby reducing the processing pressure of the service interface, reducing the impact on normal business data processing, improving the compensation efficiency of the data to be compensated, and alleviating the resource occupation problem of the compensation queue.

[0057] The data compensation method provided in this disclosure can be understood as an adaptive data compensation strategy. Its main purpose is to ensure both correct and efficient data processing during the compensation process. The overall data compensation process is not significantly different from related technologies; if data processing fails, it enters a compensation queue, and then data to be compensated is periodically retrieved from the queue for further processing. The difference lies in the operations after retrieving the data. After retrieving the data, data compensation prediction and data compensation processing are performed. Data compensation prediction can be understood as determining the current compensation failure rate of the retrieved data and judging whether the failure rate is less than a preset threshold. Data compensation processing can be understood as implementing different processing measures for the data to be compensated based on the data compensation prediction result. Specifically, if the compensation failure rate is less than the preset threshold, the data to be compensated is processed in real time; if the compensation failure rate is not less than the preset threshold, the data to be compensated is processed with a delay.

[0058] In one possible implementation, step S201 may include: retrieving the current data to be compensated and determining the compensation failure rate for the current data to be compensated. Specifically, refer to... Figure 3 Step S201 may include:

[0059] Step S2010: Retrieve the current data to be compensated;

[0060] The data to be compensated at present can be understood as the data to be compensated retrieved at the current moment.

[0061] Step S2011: Obtain the number of data processing failures within the preset time period before the current time, and the total number of data processing failures within the preset time period;

[0062] Prior to the current moment, there may have been processing of data awaiting compensation and data from normal business operations. After processing each piece of data, the type of the processing result is recorded, i.e., whether the data processing was successful or failed. The time point information corresponding to the processing result is also recorded. After each piece of data awaiting compensation is retrieved, the number of data processing failures within a preset time period prior to the current moment, and the total number of data processing attempts within that preset time period, are obtained. Specifically, based on the recorded data processing result types and corresponding time point information, the number of data processing failures and the total number of data processing attempts within the preset time period prior to the current moment can be obtained. The data in this step can include data from normal business request processing and data awaiting compensation. Data from normal business request processing can be understood as data from the initial request processing, i.e., data not in the compensation queue.

[0063] In one example, if data processing is successful, the data is marked with a success identifier; if data processing fails, the data is marked with a failure identifier. Based on these identifiers, the number of data processing failures within a preset time period and the total number of data processing attempts can be obtained.

[0064] Step S2012: The ratio of the number of data processing failures to the total number of data processing failures is used as the compensation failure rate of the current data to be compensated.

[0065] The failure rate of compensation can be obtained using the following formula:

[0066] Compensation failure rate = number of failed records / total number of requests; where the number of failed records is the number of times data processing failed within the preset time period, and the total number of requests is the total number of times data processing was performed within the preset time period.

[0067] Each failed data point is added to a compensation queue. Compensation data can be periodically retrieved from this queue, for example, every 300ms or 500ms. The data retrieved at the current moment is the current compensation data. Then, a compensation failure rate is determined for this current data. Typically, this failure rate is the ratio of the number of data failures within a preset time period to the total number of data processing iterations. The preset time period can be a period prior to the current moment or a period prior to the moment the previous data processing result was obtained. The duration of the preset time period can be set based on experience or actual needs, such as 5 minutes or 6 minutes.

[0068] After obtaining the current compensation failure rate of the data to be compensated, the failure rate is compared with a preset threshold. If it is less than the preset threshold, the next step can proceed, i.e., real-time processing of the current data to be compensated (processing immediately). If it is not less than the preset threshold, the current data to be compensated is processed with a delay (waiting for a period of time before processing). The preset threshold can be configured according to actual conditions, such as 30%, 35%, etc.

[0069] In this embodiment of the disclosure, the ratio of the number of data processing failures to the total number of data processing failures within a preset time period before the current time is used as the current compensation failure rate of the data to be compensated. This is beneficial for determining compensation measures for the data to be compensated based on the relationship between the compensation failure rate and a preset threshold.

[0070] Pulling the current data to be compensated from the compensation queue and determining the compensation failure rate of the current data to be compensated is helpful in determining the compensation measures for the current data to be compensated based on the relationship between the compensation failure rate and the preset threshold, that is, determining whether to perform real-time processing or delayed processing.

[0071] In one possible implementation, step S202 may include:

[0072] S2020: If the compensation failure rate is less than the preset threshold, the data to be compensated will be sent to the service interface corresponding to the data to be compensated in real time.

[0073] The service interface is used to process the data to be compensated.

[0074] After receiving the data to be compensated, the service interface processes it, that is, consumes the data to be compensated. Specifically, the service interface processes the data to be compensated according to the request information carried by the data to be compensated. For example, if the request information carried by the data to be compensated is to view the news list, then the service interface sends the news list to the corresponding terminal.

[0075] If the processing is successful, the data to be compensated will be recorded in the success queue; if the processing fails, the data to be compensated will be recorded in the failure queue. For data from normal business operations, the data will also be recorded in either the success queue or the failure queue depending on the type of processing result.

[0076] The service interface processes the data to be compensated in a similar manner to the steps described above, and will not be repeated here.

[0077] Step S2031: If the compensation failure rate is not less than a preset threshold, determine the delay time based on the average processing time of successful processes.

[0078] The average successful processing time (succAvgTime) represents the average time taken to successfully process data. It is calculated by recording the moment data is received at the service interface and the moment when successful processing is completed; the time difference between these two moments is the average successful processing time. The average successful processing time can be calculated by averaging the successful processing times of data processed within a preset time period. The preset time period can be customized, such as 5 minutes or 6 minutes, and can be a period preceding the current moment or a period from the start of the data compensation task to the current moment.

[0079] Step S2032: After the delay time, the data to be compensated is processed.

[0080] After the delay period, the data to be compensated is sent to the corresponding service interface, which then processes the data.

[0081] The main purpose of this step is to alleviate the processing pressure. If the current compensation failure rate is higher than the previous compensation failure rate, it means that the service interface is overloaded. If the processing pressure of the service interface is not relieved, the data to be compensated will fail to be processed and will re-enter the compensation queue to wait for processing, which will cause problems such as low compensation efficiency and the occupation of more compensation queue storage resources.

[0082] In this embodiment of the disclosure, the delay time is determined based on the average processing time of successful processing. After the delay time, the data to be compensated is then processed. This can reduce the occurrence of loop compensation failures, thereby improving the compensation efficiency of the data to be compensated and alleviating the resource occupation problem of the compensation queue.

[0083] In other embodiments, step S203 may include:

[0084] If the compensation failure rate is not less than a preset threshold, the data to be compensated will be sent to the service interface corresponding to the data to be compensated after a delay.

[0085] The service interface processes the data to be compensated in a similar manner to step S202 above, and will not be described again here.

[0086] In this embodiment of the disclosure, real-time or delayed processing of the data to be compensated serves two purposes: one is to process the data to be compensated sent to the service interface, and the other is to provide basic information for data compensation prediction. This can be understood as recording the type of data to be compensated processing results, i.e., recording successfully processed data to be compensated in a success queue and recording unprocessed data to be compensated in a failure queue. This is beneficial for calculating the compensation failure rate based on the data recorded in the success queue and the failure queue, and then comparing it with a preset threshold to determine whether to perform real-time or delayed processing.

[0087] In some embodiments, failed and successful data can be recorded in different queues. In the case of data processing failure, the failed data is recorded in the failure queue, and in the case of successful data processing, the successful data is recorded in the success queue. Figure 2 Based on the illustrated embodiment, in one possible implementation, such as Figure 4 As shown, the data compensation methods include:

[0088] Step S2010: Retrieve the current data to be compensated;

[0089] Step S401: Based on the data of processing failures recorded in the failure queue, count the number of data processing failures within a preset time period;

[0090] Step S402: Based on the successfully processed data recorded in the success queue, count the number of times the data was successfully processed within a preset time period;

[0091] Step S403: The sum of the number of data processing failures and the number of data processing successes within the preset time period is taken as the total number of data processing operations within the preset time period.

[0092] When calculating the number of failed processing attempts, the number of successful processing attempts, and the total number of processing attempts, it is only necessary to calculate the number of attempts within the most recent period, such as the last 5 minutes or 6 minutes. In other words, it is necessary to calculate the number of successful and failed processing attempts and the total number of processing attempts within the most recent preset time period. The most recent preset time period can be a preset time period before the current moment.

[0093] In this embodiment of the disclosure, the number of times data processing was successfully completed, the number of times processing failed, and the total number of processing times within a preset time period are obtained based on the success queue and the failure queue, thereby obtaining the compensation failure rate of the data to be compensated.

[0094] Step S2012: The ratio of the number of data processing failures to the total number of data processing failures is used as the compensation failure rate of the current data to be compensated.

[0095] Step S204: Record the compensation failure rate after each determination of the compensation failure rate of the data to be compensated;

[0096] In the data compensation task, after each time the data to be compensated is pulled from the compensation queue, the compensation failure rate of the pulled data is calculated and recorded. Specifically, the compensation failure rate can be recorded in a queue for use in subsequent steps.

[0097] Step S202: If the compensation failure rate is less than a preset threshold, process the data to be compensated in real time; and execute step S205.

[0098] Step S405: If the compensation failure rate is not less than a preset threshold, obtain the previous compensation failure rate and delay coefficient.

[0099] The failure rate of each piece of data to be compensated pulled from the compensation queue is recorded. The failure rate of the previous data to be compensated can be obtained from the record. Specifically, the failure rate of the previous compensation can be obtained from the queue that records the failure rate of compensation.

[0100] For each data point to be compensated with a failure rate not less than a preset threshold, calculate the average processing time for the data point to be compensated.

[0101] In one feasible implementation, during the initial delay processing of the data to be compensated in the data compensation task, the average successful processing time corresponding to the data to be compensated can be directly used as the delay time (delay coefficient is 1). After the delay time, the data to be compensated is processed. At this time, processing may still fail, meaning that the data sent to the service interface simultaneously exceeds the QPS of the service interface, resulting in high processing pressure on the service interface and causing the data to be compensated to fail. Then, new data to be compensated is fetched. Because the previous data to be compensated failed, the failure rate of the new data to be compensated is not less than the previous failure rate, indicating that the previous delay time did not alleviate the processing pressure. The delay time corresponding to the new data to be compensated needs to be increased to alleviate the processing pressure. Increasing the delay time corresponding to the new data to be compensated can be achieved by multiplying the average successful processing time corresponding to the new data to be compensated by a delay coefficient greater than 1.

[0102] Each time the delay time is determined based on the average successful processing time corresponding to a piece of data to be compensated, the delay coefficient corresponding to that piece of data can be recorded. If the average successful processing time is directly used as the delay time, the corresponding delay coefficient is 1. In this step, the previous compensation failure rate and delay coefficient can be obtained based on the records of the compensation failure rate and delay coefficient.

[0103] Step S406: If the current compensation failure rate is not less than the previous compensation failure rate, then increase the previous delay coefficient to obtain the current delay coefficient;

[0104] In one feasible implementation, the previous delay coefficient can be increased by a preset step size. For example, the previous delay coefficient can be increased by one to obtain the current delay coefficient. For instance, the delay coefficient is 1 for the first delay, and the delay coefficients for subsequent delay processing are 2, 3, 4, etc., until the current compensation failure rate is less than the previous compensation failure rate, indicating that the service interface processing pressure has been alleviated.

[0105] Step S407: Multiply the current delay coefficient by the average time taken for successful processing to obtain the delay time.

[0106] This can be understood as multiplying the current delay coefficient by the average processing time of the current data to be compensated to obtain the delay time corresponding to the current data to be compensated.

[0107] The above steps S405 to S407 are one way to determine the delay time. In other embodiments, if the compensation failure rate is not less than a preset threshold, the average time taken for successful processing can be used as the delay time.

[0108] If the failure rate of the compensation of the data to be compensated is not less than a preset threshold, calculate the average processing time of the data to be compensated, and use the average processing time of the data to be compensated as the delay time. After the delay time, the data to be compensated is processed.

[0109] In this embodiment, the average processing time for the data to be compensated is directly used as the delay time. Data processing is performed after the delay time, which can reduce the occurrence of loop failures in the cyclic compensation process. Furthermore, it can improve the compensation efficiency of the data to be compensated and alleviate the resource consumption problem of the compensation queue.

[0110] Step S2032: After the delay time, the data to be compensated is processed.

[0111] Step S205: If data processing fails, record the failed data in the failure queue; if data processing succeeds, record the successful data in the success queue.

[0112] In this embodiment of the disclosure, the delay time is determined based on the average processing time of successful processing. The data to be compensated is then processed after the delay time, which can reduce the failure of the cyclic compensation cycle, thereby improving the compensation efficiency of the data to be compensated and alleviating the resource occupation problem of the compensation queue.

[0113] The data compensation methods in related technologies lead to at least two significant problems. First, the compensation efficiency for the data to be compensated is low. Normal business data processing typically doesn't have efficiency issues; that is, the number of data sent to the service interface simultaneously does not exceed the service interface's QPS, and the data can be processed normally using traditional methods. However, if there are frequent service interface overload issues—that is, the number of data sent to the service interface exceeds the service interface's QPS, and the excess data cannot be processed in time—the traditional method will encounter problems. The most obvious issue is that after processing overload, the data to be compensated will be repeatedly processed and repeatedly enter the compensation queue, leading to timeliness issues in the processing of the data to be compensated. This results in low compensation efficiency and increases the processing pressure on the service interface, affecting the data processing of normal business operations.

[0114] In this embodiment of the invention, a data compensation prediction approach is adopted to alleviate the problem of low compensation efficiency of the data to be compensated due to service interface processing overload. This can be understood as follows: when the compensation failure rate is less than a preset threshold, no delay processing is performed; instead, the data to be compensated is processed directly, which can improve compensation efficiency in this situation. When the compensation failure rate is not less than the preset threshold, the data to be compensated is processed with a delay, which can reduce the problem of the data to be compensated repeatedly entering the compensation queue for repeated processing due to processing failures (cyclic compensation with cyclic failure), thus improving compensation efficiency in this situation.

[0115] The data compensation methods in related technologies lead to another significant problem: they consume more compensation queue storage resources. Normally, if the compensation queue is not backlogged and data can be processed normally, there is no problem with excessive compensation queue storage resources, and traditional compensation methods are sufficient. However, if the service interface is overloaded, and the traditional method is still used, data processing failures will enter the compensation queue, as will data processing failures in normal business operations. This will cause the amount of data to be compensated in the compensation queue to accumulate, thus consuming excessive compensation queue storage resources. In the embodiments of this disclosure, compensation prediction is performed on the data to be compensated before processing. This reduces the number of failed data processing entries into the compensation queue, thereby alleviating the resource consumption problem of the compensation queue and ensuring the rational use of compensation queue storage space to a greater extent.

[0116] Figure 5 A flowchart of a data compensation method provided in this disclosure embodiment is shown below. Figure 5 The data compensation process is explained below.

[0117] Step S11: Start the data compensation task;

[0118] Step S12: Determine if there is any data to be compensated;

[0119] Step S13: If not, end the data compensation task;

[0120] Step S14: If yes, repeatedly retrieve the data to be compensated;

[0121] If there is data to be compensated, the subsequent steps can be executed; otherwise, the data compensation task can be terminated directly. Steps S11 to S14 are similar to steps 1 to 4 in related technologies, and will not be described again here.

[0122] Step S15, data compensation prediction;

[0123] When the data to be compensated is retrieved, step S15 is executed. Step S15 may include:

[0124] Step S151: Calculate the compensation failure rate;

[0125] The method for calculating the compensation failure rate has been established. Figure 3 The embodiments shown are described in detail, and will not be repeated here.

[0126] Step S152: Determine whether the failure rate is greater than a preset threshold; if yes, proceed to steps S153 and S154; if no, proceed to step S155.

[0127] The preset threshold can be set according to the actual situation.

[0128] Step S153: Calculate the average processing time for successful processing (succAvgTime);

[0129] Step S154: Perform subsequent operations after multiplying (succAvgTime) by the delay coefficient.

[0130] This step allows you to obtain the delay time corresponding to the currently retrieved data to be compensated. After the delay time, the data to be compensated is processed. Executing subsequent operations can be understood as processing the data to be compensated.

[0131] Step S155: Perform subsequent operations;

[0132] Step S16, data compensation processing;

[0133] Step S16 may include:

[0134] Step S161, data processing;

[0135] This can be understood as processing the retrieved data to be compensated, including the data from the chicken washing process.

[0136] Step S162: Determine whether the processing was successful; if yes, proceed to step S163; if no, proceed to step S164.

[0137] Step S163: Record to success queue (A); decrement the amount of data to be compensated by 1;

[0138] Successfully processed data to be compensated is recorded in the success queue, where A is the queue number.

[0139] Step S164, record to failure queue (B);

[0140] Record the data to be compensated that failed to be processed into the failure queue, where B is the number of the failure queue.

[0141] Step S17: Perform data processing on data from normal business operations;

[0142] Steps S17 and S161 can be performed simultaneously.

[0143] Step S18: Determine whether the data for normal business operations has been processed successfully;

[0144] Step S19: If not, increment the data to be compensated by 1.

[0145] If data processing for normal business operations fails, the data can be recorded in the failure queue; if data processing for normal business operations succeeds, the data can be recorded in the success queue.

[0146] In related technologies, when the service interface is overloaded, it can lead to loop compensation failures, repeated failures in processing the data to be compensated, and repeated attempts to enter the compensation process. This increases the processing pressure on the service interface and consequently increases the workload for developers, requiring significant manpower. The embodiments disclosed in this disclosure can reduce the development workload, especially for applications with frequent ranking activities. For example, it can reduce manpower costs from 30 person-days to 3-5 person-days. Furthermore, it can minimize the impact on the ranking rendering interface.

[0147] Figure 6 This is a schematic diagram of a data compensation device provided in an embodiment of this disclosure, with reference to... Figure 6 The device includes:

[0148] The determination module 601 is used to determine the current compensation failure rate of the current data to be compensated. The compensation failure rate is used to represent the proportion of data processing failures relative to the total data processing.

[0149] The real-time processing module 602 is used to process the data to be compensated in real time when the compensation failure rate is less than a preset threshold.

[0150] The delay processing module 603 is used to perform delay processing on the data to be compensated when the compensation failure rate is not less than a preset threshold.

[0151] In this embodiment, the current compensation failure rate of the data to be compensated is determined. If the compensation failure rate is less than a preset threshold, the data to be compensated is processed in real time; if the compensation failure rate is not less than the preset threshold, the data to be compensated is processed with a delay. In related technologies, data to be compensated is directly processed after being retrieved from the compensation queue. When the total number of requests sent to the service interface for processing simultaneously exceeds the QPS of the service interface, the processing of the data to be compensated will fail. The failed data to be compensated will re-enter the compensation queue as new data to be compensated, and after another processing failure, it will be retrieved from the compensation queue again for processing, which may fail again. Therefore, there may be a situation of cyclical compensation failure. Compared with the prior art, this embodiment takes different measures depending on whether the compensation failure rate of the data to be compensated is less than the preset threshold. When the compensation failure rate is not less than the preset threshold, the data to be compensated is processed with a delay, which can alleviate the situation where the total number of requests sent to the service interface for processing simultaneously exceeds the QPS of the service interface, thus reducing the occurrence of cyclical compensation failure.

[0152] In one possible implementation, the determining module 601 includes:

[0153] The acquisition submodule is used to obtain the number of times data processing failed within a preset time period before the current time, as well as the total number of times data processing occurred within the preset time period.

[0154] The ratio submodule is used to calculate the ratio of the number of data processing failures to the total number of data processing failures, as the compensation failure rate.

[0155] In one possible implementation, the device further includes:

[0156] The first recording module is used to record the failed data to the failure queue when data processing fails, and to record the successful data to the success queue when data processing succeeds.

[0157] The acquisition submodule is specifically used to count the number of times data processing failed within a preset time period based on the data of processing failure recorded in the failure queue; to count the number of times data processing was successful within a preset time period based on the data of processing success recorded in the success queue; and to take the sum of the number of times data processing failed and the number of times data processing was successful within the preset time period as the total number of times data was processed within the preset time period.

[0158] In one possible implementation, the delay processing module 603 includes:

[0159] The determination submodule is used to determine the delay time based on the average time of successful processing, where the average time of successful processing represents the average time it takes to successfully process data.

[0160] The processing submodule is used to process the data to be compensated after the delay time.

[0161] In one possible implementation, a submodule is defined, specifically for using the average time taken for successful processing as the delay time.

[0162] In one possible implementation, the device further includes:

[0163] The second recording module is used to record the compensation failure rate after each determination of the compensation failure rate of the data to be compensated.

[0164] The submodule is determined, specifically for obtaining the previous compensation failure rate and delay coefficient; if the current compensation failure rate is not less than the previous compensation failure rate, the previous delay coefficient is increased to obtain the current delay coefficient; the current delay coefficient is multiplied by the average time taken for successful processing to obtain the delay time.

[0165] In one possible implementation, the real-time processing module 602 is specifically used to send the data to be compensated to the service interface corresponding to the data to be compensated in real time, and the service interface is used to process the data to be compensated.

[0166] The delay processing module 603 is specifically used to send the data to be compensated to the service interface corresponding to the data to be compensated after the delay.

[0167] In one possible implementation, the determining module 601 is specifically used to retrieve the current data to be compensated and determine the compensation failure rate for the current data to be compensated.

[0168] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0169] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0170] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0171] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0172] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0173] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the data compensation method. For example, in some embodiments, the data compensation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the data compensation method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the data compensation method by any other suitable means (e.g., by means of firmware).

[0174] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0175] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0176] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0177] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0178] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0179] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0180] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0181] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data compensation method, comprising: Determine the current compensation failure rate of the current data to be compensated, whereby the compensation failure rate represents the proportion of data processing failures relative to the total amount of data processing. If the compensation failure rate is less than a preset threshold, the data to be compensated is processed in real time. If the compensation failure rate is not less than a preset threshold, the data to be compensated is delayed. The determination of the current compensation failure rate of the current data to be compensated includes: Get the number of times data processing failed within a preset time period before the current time, and the total number of times data processing occurred within the preset time period; The ratio of the number of data processing failures to the total number of data processing failures is used as the compensation failure rate; The delay processing of the data to be compensated includes: The delay time is determined based on the average time taken for successful processing, wherein the average time taken for successful processing represents the average time taken for data processing to be successful. After the delay time, the data to be compensated is processed.

2. The method according to claim 1, further comprising: In the event of data processing failure, the failed data is recorded in the failure queue; in the event of data processing success, the successfully processed data is recorded in the success queue. The acquisition of the number of data processing failures within a preset time period prior to the current moment, and the total number of data processing operations within the preset time period, includes: Based on the data of processing failures recorded in the failure queue, count the number of data processing failures within a preset time period; Based on the successfully processed data recorded in the success queue, count the number of times the data was successfully processed within a preset time period; The sum of the number of data processing failures and the number of data processing successes within the preset time period is taken as the total number of data processing operations within the preset time period.

3. The method according to claim 1, wherein, The determination of the delay time based on the average successful processing time includes: The average time taken for successful processing is taken as the delay time.

4. The method according to claim 1, further comprising: Record the compensation failure rate after each determination of the compensation failure rate of the data to be compensated; The determination of the delay time based on the average successful processing time includes: Obtain the failure rate and delay coefficient of the previous compensation; If the current compensation failure rate is not less than the previous compensation failure rate, then increase the previous delay coefficient to obtain the current delay coefficient; The delay time is obtained by multiplying the current delay coefficient by the average time taken for successful processing.

5. The method according to any one of claims 1 to 4, wherein, The real-time processing of the data to be compensated includes: The data to be compensated is sent to the service interface corresponding to the data to be compensated in real time, and the service interface is used to process the data to be compensated. The delay processing of the data to be compensated includes: After a delay, the data to be compensated will be sent to the service interface corresponding to the data to be compensated.

6. The method according to any one of claims 1 to 4, wherein, Determining the current compensation failure rate of the current data to be compensated includes: Retrieve the current data to be compensated and determine the compensation failure rate for the current data to be compensated.

7. A data compensation device, comprising: The determination module is used to determine the current compensation failure rate of the current data to be compensated, wherein the compensation failure rate is used to represent the proportion of data processing failures relative to the total data processing. The real-time processing module is used to process the data to be compensated in real time when the compensation failure rate is less than a preset threshold. The delay processing module is used to perform delay processing on the data to be compensated when the compensation failure rate is not less than a preset threshold. The determining module includes: The acquisition submodule is used to acquire the number of times data processing failed within a preset time period before the current time, and the total number of times data processing occurred within the preset time period; The ratio submodule is used to take the ratio of the number of data processing failures to the total number of data processing failures as the compensation failure rate; The delay processing module includes: The determination submodule is used to determine the delay time based on the average time of successful processing, wherein the average time of successful processing represents the average time of successful data processing; The processing submodule is used to process the data to be compensated after the delay time.

8. The apparatus according to claim 7, further comprising: The first recording module is used to record the failed data to the failure queue when data processing fails, and to record the successful data to the success queue when data processing succeeds. The acquisition submodule is specifically used to count the number of data processing failures within a preset time period based on the data of processing failures recorded in the failure queue. Based on the successfully processed data recorded in the success queue, count the number of times the data was successfully processed within a preset time period; The sum of the number of data processing failures and the number of data processing successes within the preset time period is taken as the total number of data processing operations within the preset time period.

9. The apparatus according to claim 7, wherein, The determining submodule is specifically used to take the average time of successful processing as the delay time.

10. The apparatus according to claim 7, further comprising: The second recording module is used to record the compensation failure rate after each determination of the compensation failure rate of the data to be compensated; The determining submodule is specifically used to obtain the previous compensation failure rate and delay coefficient; if the current compensation failure rate is not less than the previous compensation failure rate, the previous delay coefficient is increased to obtain the current delay coefficient; the current delay coefficient is multiplied by the average successful processing time to obtain the delay time.

11. The apparatus according to any one of claims 7 to 10, wherein, The real-time processing module is specifically used to send the data to be compensated to the service interface corresponding to the data to be compensated in real time, and the service interface is used to process the data to be compensated. The delay processing module is specifically used to send the data to be compensated to the service interface corresponding to the data to be compensated after a delay.

12. The apparatus according to any one of claims 7 to 10, wherein, The determining module is specifically used to retrieve the current data to be compensated and determine the compensation failure rate for the current data to be compensated.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images