Delay data correction method and device, equipment and medium

Abstract

The invention discloses a late data correction method and device, equipment and a medium, and the method comprises the steps: obtaining a plurality of pieces of initial arrival data in a current statistical time window, and determining a target event corresponding to a first data identifier of each piece of initial arrival data; determining a plurality of second data identifiers corresponding to the target event, matching the first data identifiers with the second data identifiers, and determining the amount of late data of the current statistical time window based on a matching result; determining statistical delay duration associated with the delay data quantity, and determining supplementary statistical time of the current statistical time window according to window closing time of the current statistical time window and the statistical delay duration; and obtaining supplementary arrival data in the supplementary statistical time, and carrying out aggregation processing on the supplementary arrival data and the initial arrival data to obtain a delayed data correction result of the current statistical time window. Through the technical scheme, the late data correction efficiency and the correction result accuracy can be improved.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and specifically relates to a method, apparatus, device and medium for correcting late data. Background Technology

[0002] Delayed data is data that arrives at the data consumer after the expected statistical time window due to network latency, equipment malfunction, system congestion, or other reasons. With the widespread application of data acquisition and processing technologies, ensuring data consistency and timeliness has become a basic requirement in systems with high real-time requirements. The existence of delayed data can lead to inconsistencies between the data consumption system and the data transmission system. Therefore, timely correction of delayed data has become crucial.

[0003] In existing technologies, the correction of late data mainly relies on historical data reception. A time series model is pre-trained, and based on the actual data received within each statistical time window, the model predicts late data for that window. The predicted late data is then added to the aggregated results for the corresponding statistical time window for late data correction. However, data transmission is random, leading to inaccurate correction results when using existing technologies. Furthermore, prior to late data correction, existing technologies require training numerous time series models, resulting in cumbersome correction steps and low efficiency. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, device, and medium for correcting late data, which solves the problems of low efficiency and inaccurate correction results in the prior art. By matching the data identifiers of the target events corresponding to each initial arrival data in the current statistical time window, the number of late data and the statistical delay duration in the current statistical time window are determined. Supplementary arrival data is obtained based on the statistical delay duration, and late data correction is performed on the current statistical time window based on the supplementary arrival data. This achieves the goal of late data supplementation correction without training a time series model, thereby improving the efficiency of late data correction and the accuracy of the correction results.

[0005] In a first aspect, embodiments of this application provide a method for correcting late data, the method comprising: Obtain multiple initial arrival data points within the current statistical time window, and determine the target event corresponding to the first data identifier of each initial arrival data point; Identify multiple second data identifiers corresponding to the target event, match the first data identifier with the second data identifiers, and determine the number of late data in the current statistical time window based on the matching results; Determine the statistical delay duration associated with the amount of late data, and determine the supplementary statistical time for the current statistical time window based on the window closing time and statistical delay duration of the current statistical time window; Obtain supplementary arrival data within the supplementary statistical time period, and aggregate the supplementary arrival data and the initial arrival data to obtain the late data correction result for the current statistical time window.

[0006] Furthermore, the supplementary arrival data and the initial arrival data are aggregated, including: Obtain the target statistical information and the first aggregation result for the current statistical time window; The initial arrival data and supplementary arrival data are aggregated according to the target statistics to obtain the second aggregation result for the current statistical time window. Based on the first aggregation result, the aggregation state of the second aggregation result is updated and verified. If the aggregation state update verification passes, the second aggregation result replaces the first aggregation result.

[0007] Furthermore, based on the first aggregation result, the aggregation state of the second aggregation result is updated and verified, including: The second aggregation result is encoded based on a preset encoding rule to obtain the second aggregation sequence; Obtain the first aggregation sequence corresponding to the first aggregation result, and determine whether the first aggregation sequence is the same as the second aggregation sequence and whether the first aggregation sequence is adjacent to the second aggregation sequence; If the first aggregation sequence and the second aggregation sequence are different but adjacent, the aggregation state update verification of the second aggregation result is passed.

[0008] Furthermore, the second aggregation sequence includes a second aggregation version, and the first aggregation sequence includes a first aggregation version; When the first aggregation sequence and the second aggregation sequence are different but adjacent, the aggregation state update verification of the second aggregation result is passed, including: If the first aggregation version is different from the second aggregation version, and the version number of the first aggregation version is adjacent to the version number of the second aggregation version, then the aggregation status update verification of the second aggregation result is determined to be successful.

[0009] Furthermore, the first data identifier is matched with the second data identifier, and the number of late data in the current statistical time window is determined based on the matching result, including: Construct a second identifier location mask for the second data identifier, and construct a first identifier location mask for the first data identifier according to the mask template of the second identifier location mask; Calculate the degree of mask difference between the first and second identifier location masks, and determine the number of late data in the current statistical time window based on the degree of mask difference.

[0010] Furthermore, the degree of mask difference between the first identifier location mask and the second identifier location mask is calculated, including: The first identifier position mask is compressed based on the first identifier parameters at each position of the mask template to obtain the first mask length and the second mask length of the second identifier position mask. Calculate the length difference between the first mask length and the second mask length to obtain the degree of mask difference between the first identifier position mask and the second identifier position mask.

[0011] Furthermore, before acquiring multiple initial arrival data points within the current statistical time window, the method also includes: Read the data occurrence time and actual consumption time of each data item in the data consumption log, and determine the standard consumption time of each data item based on the data occurrence time and the preset data duration threshold. If the standard consumption time is less than the actual consumption time, late data is detected in the data consumption log, triggering a late data alarm.

[0012] Secondly, embodiments of this application provide a late data correction device, the device comprising: The target event identification module is used to acquire multiple initial arrival data in the current statistical time window and determine the target event corresponding to the first data identifier of each initial arrival data. The late data quantity determination module is used to determine multiple second data identifiers corresponding to the target event, match the first data identifier with the second data identifier, and determine the number of late data in the current statistical time window based on the matching result; The statistical time determination module is used to determine the statistical delay duration associated with the number of late data, and to determine the supplementary statistical time for the current statistical time window based on the window closing time and statistical delay duration of the current statistical time window. The late data correction module is used to obtain supplementary arrival data within the supplementary statistical time period, and to aggregate the supplementary arrival data and the initial arrival data to obtain the late data correction result for the current statistical time window.

[0013] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0014] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0015] Fifthly, embodiments of this application also provide a computer program product comprising a computer program stored in a computer-readable storage medium, wherein at least one processor of the device reads from the computer-readable storage medium and executes the computer program, causing the device to perform the method described in the first aspect.

[0016] In this embodiment, multiple initial arrival data points within the current statistical time window are obtained, and a target event corresponding to a first data identifier for each initial arrival data point is determined. Multiple second data identifiers corresponding to the target events are determined, and the first and second data identifiers are matched. Based on the matching results, the number of late data points within the current statistical time window is determined. The statistical delay duration associated with the number of late data points is determined, and the supplementary statistical time for the current statistical time window is determined based on the window closing time and the statistical delay duration. Supplementary arrival data within the supplementary statistical time is obtained, and the supplementary arrival data and initial arrival data are aggregated to obtain the late data correction result for the current statistical time window. This late data correction method solves the problems of low efficiency and inaccurate correction results in existing technologies. By matching the data identifiers of the target events corresponding to each initial arrival data point within the current statistical time window, determining the number of late data points and the statistical delay duration within the current statistical time window, obtaining supplementary arrival data based on the statistical delay duration, and performing late data correction on the current statistical time window based on the supplementary arrival data, the goal of late data supplementation and correction without training a time series model can be achieved, improving the efficiency of late data correction and the accuracy of the correction results. Attached Figure Description

[0017] Figure 1 This is a flowchart of a method for correcting late data provided in an embodiment of this application; Figure 2 This is a diagram illustrating the supplementary statistical time window provided in this application; Figure 3 This is a flowchart of the aggregated arrival data provided in the embodiments of this application; Figure 4 This is a flowchart of the process for determining delayed data provided in an embodiment of this application; Figure 5 This is a structural block diagram of a late data correction device provided in an embodiment of this application; Figure 6 This is a structural block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application are described in detail below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0019] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0020] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0021] First, this solution can be used in scenarios involving the correction of late data, particularly in situations where delayed data consumption systems need to supplement the reception of late-arriving data to ensure data consistency. By matching the data identifiers of the target events corresponding to each initially arriving data within the current statistical time window, the number of late data points and the statistical delay duration within the current statistical time window are determined. Supplementary arriving data is then obtained based on the statistical delay duration, and late data correction is performed on the current statistical time window based on this supplementary arriving data. This achieves the goal of late data supplementation and correction without training a time series model, improving the efficiency and accuracy of late data correction. Based on the above use case, it is understandable that the implementing entity of this solution can be a server.

[0022] The following description, in conjunction with the accompanying drawings, details a method, apparatus, device, and medium for correcting delayed data provided in this application, through specific embodiments and application scenarios.

[0023] Figure 1 This is a flowchart of a method for correcting delayed data provided in an embodiment of this application. Figure 1 As shown, the specific steps include the following: S101, obtain multiple initial arrival data in the current statistical time window, and determine the target event corresponding to the first data identifier of each initial arrival data.

[0024] The current statistical time window can be the data statistical period corresponding to the current moment. Multiple time windows of equal length can be preset to statistically analyze the pending data received by the consumption system. For example, if the consumption system is a game recharge processing system, when counting the number of recharges or the recharge amount per hour, one hour is the length of one time window. Each time window includes a window opening time and a window closing event; the time interval between the window opening and closing times is the window length. Initial arriving data can be data that arrives on time within the current statistical time window. For example, if the current statistical time window is 10:00-11:00, and data received within the current statistical time window occurs at 10:05, then this data is considered on-time arriving data, i.e., an initial arriving data within the current statistical time window. The first data identifier can be a marker indicating the meaning of each initial arriving data and the originating point of the data. The first data identifier can be automatically generated by the data sender based on the operation corresponding to each data item and the corresponding client. The target event can be the pending business request from the client corresponding to the arriving data received within the current statistical time window. For example, if the arrived data includes information such as the order ID, payment amount, order status parameters, and payment method parameters issued by client A, then the first data identifiers corresponding to the above data are the order name, payment amount, order status, and payment method issued by client A, respectively. The target event corresponding to the first data identifier is the payment transaction requested by client A. Since each piece of data is generated based on operations from the same or different clients, the arrived data in the current statistical time window can correspond to target events from multiple different clients.

[0025] In one embodiment, all data within the current statistical time window can be read, and the occurrence time of each data entry can be matched with the current statistical time window. Data falling within the current statistical time window range is used as initial arriving data, resulting in multiple initial arriving data entries. The data occurrence time can be the time when the data was generated. For example, the occurrence time of the order ID data is the time when the order system generates the order number after receiving the order instruction. The data occurrence time is carried during data transmission and consumption. After the data arrives at the consumer end, it is associated with the data's first data identifier and recorded in the data consumption log. The first data identifier corresponding to each initial arriving data entry can be read, and the target event corresponding to each first data identifier can be determined based on the correspondence between the first data identifier and the pending business.

[0026] In one embodiment, before acquiring multiple initial arrival data within the current statistical time window, the method further includes: reading the data occurrence time and actual consumption time of each data entry in the data consumption log; determining the standard consumption time of each data entry based on the data occurrence time and a preset data arrival duration threshold; and determining that late data exists in the data consumption log if the standard consumption time is less than the actual consumption time, triggering a late data alarm.

[0027] The data consumption log records information about the data consumption system's operations, including reading, storing, and consuming data. It records information such as the data ID, data volume, consumption time, and success or failure of each data entry, as well as the data's own occurrence time and first data identifier. The actual consumption time is the time the data consumption system actually processes each data entry. In this solution, the actual time the data consumption system receives each data entry is considered the actual consumption time. The preset data arrival time threshold is the maximum time data takes to reach the data consumption system under normal transmission conditions (no delays or congestion). The standard consumption time is the time it takes for data to arrive at the data consumption system under normal transmission conditions, calculated based on its occurrence time and the preset data arrival time threshold. Late data is data that arrives at the data consumption system delayed due to the absence of delays or congestion during transmission.

[0028] In one embodiment, before acquiring multiple initial arrival data within the current statistical time window, the data occurrence time and actual consumption time of each data item recorded in the data consumption log can be read. The standard consumption time of each data item can be calculated based on the data occurrence time and a preset data arrival duration threshold. The relationship between the standard consumption time and the actual consumption time can be compared. If the standard consumption time is less than the actual consumption time, it can be determined that there is late data in the data consumption log, and a late data alarm can be triggered to ensure the consistency of the data statistical results within the current statistical time window.

[0029] This solution reads the occurrence time and actual consumption time of each data entry in the data consumption log, determines the standard consumption time for each data entry based on the occurrence time and a preset data duration threshold, and triggers a late data alarm if late data is detected in the data consumption log. This achieves the goal of real-time monitoring of late data based on the data consumption log, ensuring the timeliness of late data detection and avoiding data consistency issues.

[0030] S102, determine multiple second data identifiers corresponding to the target event, match the first data identifier with the second data identifier, and determine the number of late data in the current statistical time window based on the matching result.

[0031] The second data identifier can be the data identifier of each of the multiple data entries corresponding to the target event. The second data identifier can include the first data identifier that has arrived in the current statistical time window, and can also include other data identifiers that are delayed in arrival.

[0032] In one embodiment, multiple second data identifiers corresponding to the target event can be determined based on the correspondence between the target event and the data identifier. The first data identifier is compared with the second data identifiers to determine the data identifiers that exist in the first data identifiers but do not exist in the first data identifiers. These data identifiers are used as the data identifiers of late data. The data of the data identifiers of late data is counted to determine the number of late data in the current statistical time window.

[0033] S103, determine the statistical delay duration associated with the number of late data, and determine the supplementary statistical time for the current statistical time window based on the window closing time and statistical delay duration of the current statistical time window.

[0034] The statistical delay duration can be the extended period of time the statistical time window is closed. The supplementary statistical time can be the time during which data is received and statistically analyzed again after the current statistical time window has closed.

[0035] In one embodiment, the statistical delay duration associated with the number of late data points can be determined based on the correlation between the number of late data points and the statistical delay duration. The more late data points, the longer the statistical delay duration. Supplementary statistical time after the window closing time is determined based on the current statistical time window's closing time and the statistical delay duration.

[0036] Figure 2 This is a schematic diagram of the supplementary statistical time window provided in this application.

[0037] like Figure 2 As shown, the data consumer receives data sent by the data sender in real time. Based on a preset statistical duration, the timeline is divided into multiple statistical time windows, such as statistical time window A, statistical time window B, and statistical time window C, each with a length of one hour. For statistical time window A, the number of late data points can be determined based on the target event corresponding to the first data identifier of each initially arriving data point within that window. Then, the statistical delay duration of statistical time window A is determined based on the correlation between the number of late data points and the statistical delay duration. A supplementary statistical time A1 is obtained based on the window closing time and statistical delay duration of statistical time window A. Data received within the time range corresponding to supplementary statistical time A1, belonging to statistical time window A, can still participate in the aggregation calculation of data within statistical time window A, avoiding consistency issues between the data sender and data consumer caused by data delays at the data sender.

[0038] S104: Obtain supplementary arrival data within the supplementary statistical time period, and aggregate the supplementary arrival data and the initial arrival data to obtain the late data correction result for the current statistical time window.

[0039] Specifically, supplementary arriving data refers to data that arrives within the supplementary statistical time window and belongs to the current statistical time window. Aggregation processing can be the process of combining and statistically analyzing all data belonging to the current statistical time window. Late data correction results can be the statistical results of the current statistical time window after adding late data.

[0040] In one embodiment, data received by the data consumption system within the supplementary statistical time period can be acquired, and supplementary arriving data can be determined based on whether the occurrence time of each data point falls within the time range of the current statistical time window. The supplementary arriving data is combined with the initial arriving data to form the data for the current statistical time window, thus obtaining the late data correction result for the current statistical time window.

[0041] The technical solution provided in this application embodiment obtains multiple initial arrival data in the current statistical time window, determines the target event corresponding to the first data identifier of each initial arrival data, determines multiple second data identifiers corresponding to the target event, matches the first data identifiers with the second data identifiers, and determines the number of late data in the current statistical time window based on the matching result; determines the statistical delay duration associated with the number of late data, and determines the supplementary statistical time of the current statistical time window based on the window closing time and statistical delay duration; obtains supplementary arrival data within the supplementary statistical time, and aggregates the supplementary arrival data and initial arrival data to obtain the late data correction result for the current statistical time window. This late data correction method solves the problems of low efficiency and inaccurate correction results in the prior art. By matching the data identifiers of the target events corresponding to each initial arrival data in the current statistical time window, determining the number of late data and statistical delay duration of the current statistical time window, obtaining supplementary arrival data based on the statistical delay duration, and performing late data correction on the current statistical time window based on the supplementary arrival data, the goal of late data supplementation correction without training a time series model can be achieved, improving the efficiency and accuracy of late data correction.

[0042] Figure 3 This is a flowchart of the aggregated arrival data provided in an embodiment of this application. For example... Figure 3 As shown, the specific steps include the following: S301, obtain the target statistical information and the first aggregation result for the current statistical time window.

[0043] The target statistical information can be information describing the aggregated content of the current statistical time window. For example, it could be the number of orders placed within each hour. The first aggregation result can be the aggregated result obtained by combining and statistically analyzing the initial arrival data within the current statistical time window.

[0044] In one embodiment, target statistics and a first aggregation result for the current statistical time window can be obtained. At the window closing time of the current statistical time window, the received initial arrival data can be aggregated based on pre-stored target statistics to obtain the first aggregation result.

[0045] S302, aggregate the initial arrival data and supplementary arrival data according to the target statistics to obtain the second aggregation result of the current statistical time window.

[0046] The second aggregation result can be an aggregation result obtained by combining the initial arrival data and the supplementary arrival data and performing statistics according to the target statistical information.

[0047] In one embodiment, the initial arrival data and supplementary arrival data can be aggregated according to the target statistical information to obtain the second aggregation result of the current statistical time window.

[0048] S303, perform an aggregation state update verification on the second aggregation result based on the first aggregation result, and replace the first aggregation result with the second aggregation result if the aggregation state update verification passes.

[0049] The aggregation state update verification can be an operation that checks whether the second aggregation result is the same as the first aggregation result. Aggregate state update verification can avoid the problem of duplicate writing of aggregation results to downstream systems, effectively reducing the write overhead of downstream systems.

[0050] In one embodiment, the aggregation status update can be verified by comparing whether the first aggregation result and the second aggregation result are completely identical. If they are completely identical, it is determined that there is no new supplementary arrival data in the second aggregation result, and in this case, there is no need to update the first aggregation result, and the aggregation status update verification fails. If they are not completely identical, it is determined that there is new supplementary arrival data in the second aggregation result, and in this case, the first aggregation result needs to be updated, and the aggregation status update verification passes. If the aggregation status update verification passes, the second aggregation result replaces the first aggregation result and is rewritten into the aggregation result of the current statistics window.

[0051] In one embodiment, performing an aggregation state update verification on a second aggregation result based on a first aggregation result includes: encoding the second aggregation result based on a preset encoding rule to obtain a second aggregation sequence; obtaining a first aggregation sequence corresponding to the first aggregation result; determining whether the first aggregation sequence and the second aggregation sequence are the same, and whether the first aggregation sequence and the second aggregation sequence are adjacent; and determining that the aggregation state update verification of the second aggregation result passes if the first aggregation sequence and the second aggregation sequence are different but adjacent.

[0052] The preset encoding rule can be a rule that encodes and sorts each field in the aggregation result to uniquely represent the aggregation result. The second aggregation sequence can be an encoding sequence obtained by encoding each field in the second aggregation result and then combining them according to the field order. The first aggregation sequence can be an encoding sequence obtained by encoding each field in the first aggregation result and then combining them according to the field order. Since the multiple data received in the current statistical time window may be sent by different clients, and the current statistical time window has target statistical information, data related to the target statistical information can be filtered out first. Then, the multiple data received can be grouped according to the client corresponding to the data. The aggregation result of the data in each group is used as a field in the overall aggregation result of the current statistical time window. Each field is encoded according to the preset encoding rule, and the field encoding results corresponding to multiple clients are sorted and combined according to the arrival time of the earliest data among the multiple data corresponding to each client that arrives in the current statistical time window to obtain the aggregation sequence.

[0053] In one embodiment, the second aggregation result can be encoded according to a preset encoding rule to obtain a second aggregation sequence, and a pre-stored first aggregation sequence obtained by encoding the first aggregation result according to the preset encoding rule can be retrieved. The similarity between the first and second aggregation sequences can be determined based on their lengths and whether the sequence values ​​corresponding to the two aggregation sequences at the same sequence position are completely identical. Since supplementary arriving data can be aggregated with initial arriving data multiple times within a preset time interval within the statistical delay period, the aggregation timestamp of each aggregation operation can be recorded. By comparing the time interval between the aggregation timestamps corresponding to the first and second aggregation sequences to see if it equals a preset time interval, it can be determined whether the first and second aggregation sequences are adjacent. If the time interval between the aggregation timestamps equals the preset time interval, then the first and second aggregation sequences are determined to be adjacent. If the first and second aggregation sequences are different but adjacent, the aggregation state update verification of the second aggregation result is determined to be successful.

[0054] This scheme encodes the aggregation results and determines whether the first aggregation sequence and the second aggregation sequence are the same and adjacent. It then determines whether the aggregation state update verification of the second aggregation result passes. This achieves the purpose of performing aggregation state update verification based on the aggregation results and aggregation order, avoiding the system overhead caused by repeated writing of aggregation results and improving the accuracy of the final written aggregation results.

[0055] In one embodiment, the second aggregation sequence includes a second aggregation version, and the first aggregation sequence includes a first aggregation version; when the first aggregation sequence and the second aggregation sequence are different and adjacent, determining that the aggregation status update verification of the second aggregation result has passed includes: when the first aggregation version and the second aggregation version are different, and the version number of the first aggregation version is adjacent to the version number of the second aggregation version, determining that the aggregation status update verification of the second aggregation result has passed.

[0056] The aggregation version can be a code number obtained by encoding the aggregation result. Whenever the aggregation result changes, i.e., when aggregation is re-performed, a new version number is generated for the new aggregation version. The second aggregation version can be the version number corresponding to the second aggregation result. The first aggregation version can be the version number corresponding to the first aggregation result.

[0057] In one embodiment, the first aggregation version and the second aggregation version can be compared to see if they are the same and if their version numbers are adjacent. If the first aggregation version and the second aggregation version are different and the version numbers of the first aggregation version and the second aggregation version are adjacent, the aggregation status update verification of the second aggregation result is determined to be successful.

[0058] This solution improves the efficiency and accuracy of verifying the aggregation status update of aggregation results by determining that the aggregation status update verification of the second aggregation result is successful when the first aggregation version and the second aggregation version are different and the version numbers of the first aggregation version and the second aggregation version are adjacent.

[0059] The technical solution provided in this application embodiment obtains the target statistical information and the first aggregation result of the current statistical time window, performs aggregation processing on the initial arrival data and the supplementary arrival data to obtain the second aggregation result, performs aggregation status update verification on the second aggregation result based on the first aggregation result, and replaces the first aggregation result with the second aggregation result, which can improve the accuracy of aggregation update and thus ensure data consistency.

[0060] Figure 4 This is a flowchart illustrating the determination of delayed data provided in an embodiment of this application. For example... Figure 4 As shown, the specific steps include the following: S401, construct a second identifier location mask for the second data identifier, and construct a first identifier location mask for the first data identifier according to the mask template of the second identifier location mask.

[0061] The second identifier location mask can be a sequence obtained by arranging multiple second identifiers corresponding to the same target event in a random order and then masking the identifiers at each location. The mask template can be a template representing the location of each identifier in the second identifier location mask and the positional relationship between adjacent identifiers.

[0062] In one embodiment, a second identifier location mask for the second data identifier can be constructed, and the identifier name corresponding to each mask in the second identifier location mask can be identified to obtain a mask template for the second identifier location mask. The first data identifiers are arranged according to the position and order of each identifier in the mask template, and the template location masks that exist in the mask template but do not exist in the first data identifiers are set to 0 to obtain the first identifier location mask.

[0063] S402, calculate the degree of mask difference between the first identifier location mask and the second identifier location mask, and determine the number of late data in the current statistical time window based on the degree of mask difference.

[0064] In one embodiment, the similarity between the first identifier location mask and the second identifier location mask can be calculated, and the reciprocal of the similarity can be used as the degree of mask difference between the first identifier location mask and the second identifier location mask. The number of late data in the current statistical time window can be determined based on the correspondence between the degree of mask difference and the number of late data.

[0065] In one embodiment, calculating the mask difference between the first identifier location mask and the second identifier location mask includes: performing mask compression on the first identifier location mask based on the first identifier parameters at each position of the mask template to obtain the first mask length and obtaining the second mask length of the second identifier location mask; calculating the length difference between the first mask length and the second mask length to obtain the mask difference between the first identifier location mask and the second identifier location mask.

[0066] Mask compression can be an operation that filters out mask positions where the identifier parameter is 0 from the identifier position mask. The first mask length can be the number of identifiers in the first identifier position mask. The second mask length can be the number of identifiers in the second identifier position mask.

[0067] In one embodiment, the first identifier position mask can be masked by compressing the first identifier position mask according to the first identifier parameters at each position of the mask template to obtain the first mask length and the second mask length of the second identifier position mask. The length difference between the first mask length and the second mask length can be calculated and used as the degree of mask difference between the first identifier position mask and the second identifier position mask.

[0068] This scheme obtains the first mask length by masking the first identifier location mask and the second mask length by obtaining the second identifier location mask. It then calculates the length difference between the first mask length and the second mask length to obtain the degree of mask difference between the first identifier location mask and the second identifier location mask. This simplifies the steps of calculating the degree of mask difference and improves the calculation efficiency.

[0069] The technical solution provided in this application embodiment constructs a second identifier location mask for a second data identifier and a first identifier location mask for a first data identifier according to the mask template of the second identifier location mask. Based on the degree of mask difference between the first identifier location mask and the second identifier location mask, the number of late data in the current statistical time window is determined, which can improve the accuracy of determining the number of late data and thus improve the comprehensiveness of subsequent reception of delayed data.

[0070] Figure 5 This is a structural block diagram of a late data correction device provided in an embodiment of this application. Figure 5 As shown, it specifically includes the following: The target event identification module 501 is used to acquire multiple initial arrival data in the current statistical time window and determine the target event corresponding to the first data identifier of each initial arrival data. The late data quantity determination module 502 is used to determine multiple second data identifiers corresponding to the target event, match the first data identifier with the second data identifier, and determine the number of late data in the current statistical time window based on the matching result; The statistical time determination module 503 is used to determine the statistical delay duration associated with the number of late data, and to determine the supplementary statistical time for the current statistical time window based on the window closing time and statistical delay duration of the current statistical time window. The late data correction module 504 is used to obtain supplementary arrival data within the supplementary statistical time period, and to aggregate the supplementary arrival data and the initial arrival data to obtain the late data correction result for the current statistical time window.

[0071] Furthermore, the delayed data correction module 504 is specifically used for: Obtain the target statistical information and the first aggregation result for the current statistical time window; The initial arrival data and supplementary arrival data are aggregated according to the target statistics to obtain the second aggregation result for the current statistical time window. Based on the first aggregation result, the aggregation state of the second aggregation result is updated and verified. If the aggregation state update verification passes, the second aggregation result replaces the first aggregation result.

[0072] Furthermore, the delayed data correction module 504 is specifically used for: The second aggregation result is encoded based on a preset encoding rule to obtain the second aggregation sequence; Obtain the first aggregation sequence corresponding to the first aggregation result, and determine whether the first aggregation sequence is the same as the second aggregation sequence and whether the first aggregation sequence is adjacent to the second aggregation sequence; If the first aggregation sequence and the second aggregation sequence are different but adjacent, the aggregation state update verification of the second aggregation result is passed.

[0073] Furthermore, the second aggregation sequence includes a second aggregation version, and the first aggregation sequence includes a first aggregation version; Late data correction module 504 is specifically used for: If the first aggregation version is different from the second aggregation version, and the version number of the first aggregation version is adjacent to the version number of the second aggregation version, then the aggregation status update verification of the second aggregation result is determined to be successful.

[0074] Furthermore, the late data quantity determination module 502 is specifically used for: Construct a second identifier location mask for the second data identifier, and construct a first identifier location mask for the first data identifier according to the mask template of the second identifier location mask; Calculate the degree of mask difference between the first and second identifier location masks, and determine the number of late data in the current statistical time window based on the degree of mask difference.

[0075] Furthermore, the late data quantity determination module 502 is specifically used for: The first identifier position mask is compressed based on the first identifier parameters at each position of the mask template to obtain the first mask length and the second mask length of the second identifier position mask. Calculate the length difference between the first mask length and the second mask length to obtain the degree of mask difference between the first identifier position mask and the second identifier position mask.

[0076] Furthermore, the device also includes: The consumption time determination module is used to read the data occurrence time and actual consumption time of each data item in the data consumption log, and determine the standard consumption time of each data item based on the data occurrence time and the preset data duration threshold. The late data alarm module is used to determine the presence of late data in the data consumption log and trigger a late data alarm when the standard consumption time is less than the actual consumption time.

[0077] The technical solution provided in this application includes a target event identification module, used to acquire multiple initial arrival data in the current statistical time window and determine the target event corresponding to the first data identifier of each initial arrival data; a late data quantity determination module, used to determine multiple second data identifiers corresponding to the target event, match the first data identifier with the second data identifier, and determine the late data quantity in the current statistical time window based on the matching result; a statistical time determination module, used to determine the statistical delay duration associated with the late data quantity, and determine the supplementary statistical time of the current statistical time window based on the window closing time and statistical delay duration of the current statistical time window; and a late data correction module, used to acquire supplementary arrival data within the supplementary statistical time, and aggregate the supplementary arrival data and the initial arrival data to obtain the late data correction result for the current statistical time window. The aforementioned late data correction device solves the problems of low efficiency and inaccurate correction results in existing technologies. By matching the target events corresponding to each initial arrival data in the current statistical time window with data identifiers, the number of late data and the statistical delay duration in the current statistical time window are determined. Supplementary arrival data is obtained based on the statistical delay duration, and late data correction is performed on the current statistical time window based on the supplementary arrival data. This achieves the goal of late data supplementation and correction without training a time series model, thus improving the efficiency of late data correction and the accuracy of the correction results.

[0078] The late data correction device in this application embodiment can be configured in a device, or in a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0079] The late data correction device in this application embodiment can be an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0080] The late data correction device provided in this application embodiment can realize the various processes implemented in the above method embodiments, and will not be described again here to avoid repetition.

[0081] like Figure 6 As shown, this application embodiment also provides an electronic device 600, including a processor 601, a memory 602, and a program or instructions stored in the memory 602 and executable on the processor 601. When the program or instructions are executed by the processor 601, they implement the various processes of the above-described late data correction method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0082] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0083] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described method for correcting late data and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0084] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0085] This application also provides a program product including program code. When the program product is run on a computer device, the program code causes the computer device to perform the steps of the methods described above according to various exemplary embodiments of this application. For example, the computer device can perform a method for correcting delayed data as described in an embodiment of this application. The program product can be implemented using any combination of one or more readable media.

[0086] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0087] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0088] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0089] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the claims.

Claims

1. A method for correcting late data, characterized in that, The method includes: Obtain multiple initial arrival data within the current statistical time window, and determine the target event corresponding to the first data identifier of each initial arrival data; A plurality of second data identifiers corresponding to the target event are identified, the first data identifiers are matched with the second data identifiers, and the number of late data in the current statistical time window is determined based on the matching results; Determine the statistical delay duration associated with the number of late data, and determine the supplementary statistical time for the current statistical time window based on the window closing time of the current statistical time window and the statistical delay duration; Obtain the supplementary arrival data within the supplementary statistical time period, and aggregate the supplementary arrival data and the initial arrival data to obtain the late data correction result for the current statistical time window.

2. The method for correcting delayed data according to claim 1, characterized in that, The aggregation process of the supplementary arrival data and the initial arrival data includes: Obtain the target statistical information and the first aggregation result for the current statistical time window; The initial arrival data and the supplementary arrival data are aggregated according to the target statistical information to obtain the second aggregation result of the current statistical time window; Based on the first aggregation result, the aggregation status of the second aggregation result is updated and verified. If the aggregation status update verification passes, the second aggregation result replaces the first aggregation result.

3. The method for correcting delayed data according to claim 2, characterized in that, The step of updating and verifying the aggregation status of the second aggregation result based on the first aggregation result includes: The second aggregation result is encoded based on a preset encoding rule to obtain a second aggregation sequence; Obtain the first aggregation sequence corresponding to the first aggregation result, and determine whether the first aggregation sequence is the same as the second aggregation sequence, and whether the first aggregation sequence and the second aggregation sequence are adjacent; If the first aggregation sequence is different from and adjacent to the second aggregation sequence, the aggregation state update verification of the second aggregation result is determined to be successful.

4. The method for correcting delayed data according to claim 3, characterized in that, The second aggregation sequence includes a second aggregation version, and the first aggregation sequence includes a first aggregation version; The step of determining that the aggregation state update verification of the second aggregation result passes when the first aggregation sequence and the second aggregation sequence are different and adjacent includes: If the first aggregation version is different from the second aggregation version, and the version number of the first aggregation version is adjacent to the version number of the second aggregation version, then the aggregation status update verification of the second aggregation result is determined to be successful.

5. The method for correcting delayed data according to claim 1, characterized in that, The step of matching the first data identifier with the second data identifier and determining the number of late data in the current statistical time window based on the matching result includes: Construct a second identifier location mask for the second data identifier, and construct a first identifier location mask for the first data identifier according to the mask template of the second identifier location mask; Calculate the degree of mask difference between the first identifier location mask and the second identifier location mask, and determine the number of late data in the current statistical time window based on the degree of mask difference.

6. The method for correcting delayed data according to claim 5, characterized in that, The calculation of the mask difference between the first identifier location mask and the second identifier location mask includes: Based on the first identifier position mask at each position of the mask template, the first identifier position mask is masked to obtain the first mask length and the second identifier position mask is obtained to obtain the second mask length. Calculate the length difference between the first mask length and the second mask length to obtain the degree of mask difference between the first identifier position mask and the second identifier position mask.

7. The method for correcting delayed data according to claim 1, characterized in that, Before acquiring multiple initial arrival data points within the current statistical time window, the method further includes: Read the data occurrence time and actual consumption time of each data item in the data consumption log, and determine the standard consumption time of each data item based on the data occurrence time and the preset data duration threshold. If the standard consumption time is less than the actual consumption time, it is determined that there is late data in the data consumption log, and a late data alarm is triggered.

8. A late data correction device, characterized in that, The device includes: The target event identification module is used to acquire multiple initial arrival data in the current statistical time window and determine the target event corresponding to the first data identifier of each initial arrival data. The late data quantity determination module is used to determine multiple second data identifiers corresponding to the target event, match the first data identifier with the second data identifier, and determine the late data quantity of the current statistical time window based on the matching result; The statistical time determination module is used to determine the statistical delay duration associated with the number of late data, and to determine the supplementary statistical time of the current statistical time window based on the window closing time of the current statistical time window and the statistical delay duration. The late data correction module is used to acquire the supplementary arrival data within the supplementary statistical time period, and to aggregate the supplementary arrival data and the initial arrival data to obtain the late data correction result for the current statistical time window.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of a method for correcting late data as claimed in any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of a method for correcting late data as described in any one of claims 1-7.

Images