A method, system, terminal and medium for delaying automatic backfilling of data

Abstract

The application discloses a kind of delay data automatic backfilling method, system, terminal and medium, the method includes: whether the data uploaded by equipment is delay data;If it is delay data, calculate treatment day for delay data, and write treatment day into backfilling tracking table;Based on backfilling tracking table, directional recalculation is executed.The application can accurately identify device delay upload data, automatically account corresponding treatment day and enter backfilling tracking table, directional recalculation is realized relying on form, and medical statistical result is finally consistent under the premise of not doing full recalculation.

Description

Technical Field

[0001] This invention relates to the field of medical Internet of Things (IoT) data processing, and in particular to a method, system, terminal, and medium for automatic backfilling of delayed data. Background Technology

[0002] In the context of digital transformation in smart healthcare and the large-scale access of massive medical IoT terminals, building a stable and accurate daily data statistics and calculation system is the core support for realizing quantitative analysis of equipment usage status, medical resource scheduling, and equipment operation and maintenance assessment. It is also the foundation for ensuring consistent statistical standards and reliable accounting results for medical data. The daily data statistics layer is responsible for summarizing key values ​​such as daily equipment operation indicators and calculating the median of usage data. The statistical results can be directly used for multiple core business operations such as equipment wear and tear assessment, patient usage time analysis, and platform operation accounting. The accuracy of the statistical values ​​directly affects subsequent medical business decisions and data compliance archiving. However, in real online operation scenarios, medical terminals often experience network fluctuations, network outages, and message retransmission retries. A large amount of business data that should belong to historical dates arrives at the platform with delays. After the platform completes the daily statistics writing in advance, the delayed historical data cannot synchronously correct the generated statistical results, directly causing distortion and errors in historical daily statistical values ​​and undermining the reliability of the entire statistical data chain. To address the statistical bias caused by lagging data, three conventional correction methods have evolved within the industry, but all have significant shortcomings: First, full recalculation incurs enormous computing power costs, and the computational pressure worsens dramatically as equipment and historical data volumes continue to expand, making platform resource costs difficult to control. Second, manual recalculation relies heavily on the experience of operations and maintenance personnel to judge the range of lagging data; human intervention is prone to omissions in corrections, and the correction response is slow, failing to meet real-time statistical needs. Third, single-point event-triggered correction solutions have cumbersome overall business chain design; a natural time window exists between data entry and statistical calculation processes, still failing to completely eliminate statistical bias. Therefore, existing conventional solutions cannot simultaneously balance computational cost, automation capabilities, and statistical consistency, exhibiting significant technical limitations.

[0003] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a method, system, terminal and medium for automatic backfilling of delayed data, in response to the above-mentioned defects of the prior art. It aims to solve the problem that the existing solutions for full recalculation, manual recalculation and single-point event triggering of delayed historical data generated by network outages and message retry in medical IoT platforms have high computing power costs, are prone to omissions due to reliance on manual labor and poor timeliness, and have statistical window deviations due to complex links, making it difficult to ensure accurate and consistent daily statistical results in a low-cost and automated manner.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a method for automatically backfilling delayed data, wherein the method includes: Determine if the data uploaded by the device is delayed; If the data is delayed, calculate the treatment date for the delayed data and write the treatment date into the backfill tracking table; Based on the backfill tracking table, perform targeted recalculation.

[0006] In one implementation, determining whether the data uploaded by the device is delayed data includes: Compare the hour to which the data time belongs with the current job processing hour window; If the hour to which the data belongs is earlier than the current job processing hour window, it is identified as delayed data.

[0007] In one implementation, the targeted recalculation and regular statistics are run in parallel within the same AWS Glue daily scheduled job.

[0008] In one implementation, the treatment day is divided into a time interval from 12:00 noon on the current day to 12:00 noon on the next day.

[0009] In one implementation, when calculating the treatment date, parallel calculations are performed using both UTC+8 and UTC+9 time zone calibers.

[0010] In one implementation, the backfill tracking table includes three fields: device identifier, treatment date, and time zone.

[0011] In one implementation, performing targeted recalculation includes: If the recalculation is successful, the corresponding tracking records in the backfill tracking table will be cleared. If recalculation fails, the corresponding tracking record in the backfill tracking table will be retained to enable automatic retry.

[0012] Secondly, embodiments of the present invention also provide an automatic backfilling system for delayed data, wherein the system includes: The judgment module is used to determine whether the data uploaded by the device is delayed data; The treatment day calculation module is used to calculate the treatment day for delayed data and write the treatment day into the backfill tracking table. The targeted recalculation module is used to perform targeted recalculation based on the backfill tracking table.

[0013] Thirdly, embodiments of the present invention also provide a terminal, wherein the terminal includes a memory, a processor, and a delayed data auto-filling program stored in the memory and executable on the processor, wherein when the processor executes the delayed data auto-filling program, it implements the steps of the delayed data auto-filling method described in any of the above schemes.

[0014] Fourthly, embodiments of the present invention also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a delayed data autofill program, and when the delayed data autofill program is executed by a processor, it implements the steps of the delayed data autofill method described in any of the above schemes.

[0015] Beneficial Effects: This invention provides an automatic backfilling method for delayed data. Compared with existing technologies, this invention first determines whether the data uploaded by the device is delayed data, accurately distinguishing between regular real-time data and delayed data. This effectively filters out abnormally uploaded lagging data, preventing delayed data from being mixed with real-time data in calculations. It avoids data timing errors and statistical distortions from the source, ensuring the accuracy of subsequent data processing. Next, if it is delayed data, the treatment day is calculated and written into the backfilling tracking table. By uniformly recording and retaining the core correction information of delayed data, standardized and visual tracking management of all delayed data is achieved, eliminating problems such as data omission, missing information, and difficulty in tracing the source. This provides complete and reliable data support for subsequent data recalculation. Then, based on the backfilling tracking table… Targeted recalculation can accurately pinpoint the range of delayed data requiring correction, abandoning the redundant mode of traditional overall recalculation. It performs targeted correction calculations only on lagging and abnormal data, significantly improving the efficiency and accuracy of data recalculation. It effectively corrects statistical deviations caused by delayed data uploads, ensuring the accuracy and completeness of equipment data statistics results. Overall, this invention solves industry problems such as data statistics errors, data traceability difficulties, and low recalculation efficiency caused by delayed equipment data uploads through a closed-loop processing logic of first identifying delayed data, then standardizing and recording corrected data, and finally performing targeted and precise recalculation. It achieves standardized, refined, and efficient processing of delayed equipment data, comprehensively improving the reliability and rigor of equipment data statistical analysis, and effectively ensuring the normal operation of subsequent business statistics and analysis work based on equipment data. Attached Figure Description

[0016] Figure 1 A flowchart illustrating a specific implementation of the automatic backfilling method for delayed data provided in this embodiment of the invention.

[0017] Figure 2 This is a schematic diagram of the automatic backfill system for delayed data provided in an embodiment of the present invention.

[0018] Figure 3 This is a block diagram illustrating the internal structure of a terminal provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0020] In the context of digital transformation in smart healthcare and the large-scale access of massive medical IoT terminals, building a stable and accurate daily data statistics and calculation system is the core support for realizing quantitative analysis of equipment usage status, medical resource scheduling, and equipment operation and maintenance assessment. It is also the foundation for ensuring consistent statistical standards and reliable accounting results for medical data. The daily data statistics layer is responsible for summarizing key values ​​such as daily equipment operation indicators and calculating the median of usage data. The statistical results can be directly used for multiple core business operations such as equipment wear and tear assessment, patient usage time analysis, and platform operation accounting. The accuracy of the statistical values ​​directly affects subsequent medical business decisions and data compliance archiving. However, in real online operation scenarios, medical terminals often experience network fluctuations, network outages, and message retransmission retries. A large amount of business data that should belong to historical dates arrives at the platform with delays. After the platform completes the daily statistics writing in advance, the delayed historical data cannot synchronously correct the generated statistical results, directly causing distortion and errors in historical daily statistical values ​​and undermining the reliability of the entire statistical data chain. To address the statistical bias caused by lagging data, three conventional correction methods have evolved within the industry, but all have significant shortcomings: First, full recalculation incurs enormous computing power costs, and the computational pressure worsens dramatically as equipment and historical data volumes continue to expand, making platform resource costs difficult to control. Second, manual recalculation relies heavily on the experience of operations and maintenance personnel to judge the range of lagging data; human intervention is prone to omissions in corrections, and the correction response is slow, failing to meet real-time statistical needs. Third, single-point event-triggered correction solutions have cumbersome overall business chain design; a natural time window exists between data entry and statistical calculation processes, still failing to completely eliminate statistical bias. Therefore, existing conventional solutions cannot simultaneously balance computational cost, automation capabilities, and statistical consistency, exhibiting significant technical limitations.

[0021] To address the aforementioned issues, this embodiment provides an automatic backfilling method for delayed data. Specifically, this embodiment first determines whether the data uploaded by the device is delayed data, accurately distinguishing between regular real-time data and delayed data. This effectively filters out abnormally uploaded lagging data, preventing delayed data from being mixed with real-time data in calculations. This avoids data timing discrepancies and statistical distortions from the source, ensuring the accuracy of subsequent data processing. Next, if it is delayed data, the treatment day is calculated and written into the backfilling tracking table. By uniformly recording and retaining the core correction information of the delayed data, standardized and visual tracking management of all delayed data is achieved, eliminating problems such as data omissions, missing information, and difficulties in tracing the source. This provides complete and reliable data support for subsequent data recalculation. Then, based on the backfilling tracking table… By performing targeted recalculation, this invention can accurately pinpoint the range of delayed data requiring correction, abandoning the redundant mode of traditional overall recalculation. It performs targeted correction calculations only on lagging and abnormal data, significantly improving the efficiency and accuracy of data recalculation. This effectively corrects statistical biases caused by delayed data uploads, ensuring the accuracy and completeness of equipment data statistics results. Overall, this invention solves industry problems such as data statistical errors, data traceability difficulties, and low recalculation efficiency caused by delayed equipment data uploads through a closed-loop processing logic: first identifying delayed data, then standardizing and recording corrected data, and finally performing targeted and precise recalculation. It achieves standardized, refined, and efficient processing of delayed equipment data, comprehensively improving the reliability and rigor of equipment data statistical analysis, and effectively ensuring the normal operation of subsequent business statistics and analysis work based on equipment data.

[0022] The automatic data backfilling method for delayed data provided in this embodiment can be applied to smart terminals, such as... Figure 1 As shown, the specific steps include the following: Step S100: Determine whether the data uploaded by the device is delayed data.

[0023] In this embodiment, it is first determined whether the data uploaded by the device is delayed data, accurately distinguishing between regular real-time data and delayed data. This can effectively filter out abnormally uploaded lagging data, prevent delayed data from being mixed with real-time data in the calculation, avoid the problems of data timing disorder and statistical result distortion from the source, and ensure the accuracy of subsequent data processing.

[0024] Specifically, step S100 includes the following steps: Step S101: Compare the hour to which the data time belongs with the current job processing hour window; Step S102: If the hour of the data time is earlier than the current job processing hour window, it is identified as delayed data; Step S103: If the hour to which the data time belongs belongs to the current job processing hour window, it is identified as real-time data and routine statistics are performed according to the normal process.

[0025] In one implementation, this invention sets up hourly aggregation operations. Treatment data packets uploaded by devices are uniformly aggregated by AWS Glue at the hourly level. AWS Glue extracts the hour to which the data belongs for each data packet and compares and verifies this hour with the current job's processing hour window. When the data's hour matches and belongs to the current job's processing hour window, the corresponding data is determined to be real-time data and undergoes routine statistics according to a preset normal process. When the data's hour is earlier than the current job's processing hour window, the corresponding data is directly identified and marked as delayed data, and a targeted recalculation process is initiated for the delayed data. The targeted recalculation process and the routine statistical process for real-time data are deployed and run in parallel within the same AWS Glue daily scheduling job. The entire hourly aggregation architecture, which integrates time-series discrimination, data diversion, and dual-task parallel computing, can simultaneously complete real-time data aggregation and statistics and backtracking recalculation of delayed data in a single daily scheduling job. By comparing the time series of hourly windows, it can accurately classify the timeliness type of data, preventing time-series chaos from causing deviations in aggregated values. At the same time, it shares a single job scheduling resource to execute two types of computing logic in parallel, saving the scheduling and starting / stopping costs of multiple independent jobs, and continuously performing regularization, classification, and hierarchical aggregation preprocessing on massive amounts of equipment treatment data packets.

[0026] Step S200: If it is delayed data, calculate the treatment day for the delayed data and write the treatment day into the backfill tracking table.

[0027] In this embodiment, the present invention configures a delayed data recalculation processing system consisting of an AWS Glue job and a MySQL backfill tracking table. The AWS Glue job is responsible for the entire process of operation scheduling and is used to perform the entire process logic of delayed data identification, treatment day time zone calculation, registration of targets to be recalculated, day-dimensional targeted recalculation, and record cleanup after recalculation. The MySQL backfill tracking table is used to uniformly register, persistently store and manage all business targets to be recalculated, and improve the stability of cross-batch task coordination. The MySQL backfill tracking table has five key fields: id, device_serial_number, treatment_day, timezone_label, and created_at. The id field, with a BIGINT data type, serves as the unique primary key for the entire backfill tracking table and is used to identify row records. The device_serial_number field, with a VARCHAR data type, stores the unique device serial number, which is the identifier of the device to be recalculated. The treatment_day field is compatible with both DATE and VARCHAR data types and stores the calculated treatment day value. The timezone_label field, with a VARCHAR data type, is used to fill in the UTC+8 and UTC+9 time zone labels to distinguish between the two sets of statistical standards. The created_at field, with a DATETIME data type, is used to write the system registration time when the current record was inserted into the database.When an AWS Glue job identifies delayed reporting of business raw data, it first performs a precise calculation of the treatment day by time zone. Treatment days uniformly follow a fixed business time division rule: the time interval of a single treatment day is from 12:00 noon on the current day to 12:00 noon on the next day in the corresponding time zone. For the same delayed raw data, it performs independent calculations in both UTC+8 and UTC+9 time zone standards in parallel. The specific calculation logic is as follows: First, the original reporting timestamp carried by the delayed data is converted to UTC+8 and UTC+9 time zone standard times respectively. Second, the "12:00 noon on the current day to 12:00 noon on the next day" interval rules of the two time zones are used to determine the treatment day: if the converted time zone standard time is greater than or equal to 12:00 noon on the current day, then the current day is considered a treatment day; if the converted time zone standard time is less than 12:00 noon on the current day, then the previous day is considered a treatment day. This process is used to derive the corresponding treatment_day value in UTC+8 and UTC+9. The treatment_day value corresponding to the C+9 caliber; after completing the dual-timezone treatment day calculation, the information of the affected objects is written into the MySQL backfill tracking table. The core information written is the device serial number, treatment_day calculated for the corresponding time zone, and timezone label. Before writing, a deduplication query is performed using device_serial_number, treatment_day, and timezone_label as a joint unique verification condition. If a record with all three completely matching exists in the table, the insertion operation is skipped. Only when there is no matching record is the insert statement executed to add a new tracking record. A single row of data in the backfill tracking table is an independent tracking record, thereby preventing the same device, the same treatment day, and the same timezone caliber from being registered repeatedly. The entire system relies on the MySQL backfill tracking table as an intermediate buffer to handle delayed recalculation tasks. The dual time zone independent conversion of treatment days can simultaneously adapt to two independent report statistical standards, ensuring that both UTC+8 and UTC+9 business reports can synchronously complete the delayed data correction. The pre-write deduplication verification can eliminate the redundant recalculation load caused by duplicate registrations, ensuring that the Glue daily statistical task can stably, orderly, and cyclically complete the report correction work for all delayed data.

[0028] Step S300: Perform targeted recalculation based on the backfill tracking table.

[0029] In this embodiment, the present invention constructs a complete automated operation chain based on Glue scheduling capabilities and a tracking table. After the daily statistical task carried by Glue reads the tracking table, it only performs targeted recalculation on the matched targets. That is, it only recalculates the registered device-treatment day-time zone combination, without performing a full historical recalculation. Using targeted recalculation instead of full recalculation can significantly reduce computing resource consumption and operation time. After the Glue daily statistical task reads the tracking table and performs targeted recalculation, it processes the tracking records according to the calculation results. If the recalculation is successful, the corresponding tracking record is deleted; if the recalculation fails, the tracking record is retained so that the next task can automatically retry. The entire process thus forms an automatic closed loop of "identification -> registration -> recalculation -> cleanup". The setting of retaining the tracking record when the recalculation fails can reliably achieve the operation effect of automatic retry. The entire automatic closed-loop system built on Glue and the tracking table does not require manual intervention. It can autonomously correct statistical deviations caused by delayed data, accurately constrain the data range of recalculation, automatically maintain the status of tracking records, and cyclically retry abnormal calculation tasks, while simultaneously taking into account the efficiency of operation and the accuracy and reliability of statistical results.

[0030] The proposed solution can be adapted to three typical business scenarios. The first scenario is when data is re-transmitted after the device has been offline for 48 hours. It can automatically register two days of backfilled data and complete targeted recalculation in the Glue daily statistics task the next day. The second scenario is when the same device re-transmits data multiple times with delays. The tracking table will first perform deduplication on duplicate device-treatment day-time zone combinations and only perform recalculation on unique combinations to avoid repeated calculations that amplify the overall computational load. The third scenario is when some recalculations fail. The tracking records corresponding to the failed calculations will be completely retained and will be automatically retried in the subsequent scheduled Glue tasks. The whole mechanism precisely constrains the data range for recalculation, automatically maintains the status of tracking records, and cyclically retries abnormal calculation tasks, while simultaneously taking into account both operational efficiency and the accuracy and reliability of statistical results.

[0031] In summary, this embodiment first determines whether the data uploaded by the device is delayed data, accurately distinguishing between regular real-time data and delayed data. This effectively filters out abnormally uploaded lagging data, preventing delayed data from being mixed with real-time data in calculations. This avoids data timing discrepancies and statistical distortions from the source, ensuring the accuracy of subsequent data processing. Next, if the data is delayed, the treatment day is calculated and written into the backfill tracking table. By uniformly recording and retaining the core correction information of the delayed data, standardized and visual tracking management of all delayed data is achieved, eliminating problems such as data omissions, missing information, and difficulties in tracing the source. This provides complete and reliable data support for subsequent data recalculation. Finally, based on the backfill tracking table, targeted recalculation is performed, accurately identifying the data requiring... To correct the range of delayed data in calculations, this invention abandons the redundant traditional approach of overall recalculation and performs targeted correction calculations only on lagging and abnormal data. This significantly improves the efficiency and accuracy of data recalculation, effectively corrects statistical biases caused by delayed data uploads, and ensures the accuracy and completeness of equipment data statistics results. Overall, this invention solves industry problems such as data statistical errors, difficulties in data traceability, and low recalculation efficiency caused by delayed equipment data uploads through a closed-loop processing logic of first identifying delayed data, then standardizing and recording corrected data, and finally performing targeted and precise recalculation. It achieves standardized, refined, and efficient processing of delayed equipment data, comprehensively improves the reliability and rigor of equipment data statistical analysis, and effectively ensures the normal operation of subsequent business statistics and analysis work based on equipment data.

[0032] like Figure 2 As shown in the illustration, this embodiment also provides an automatic backfilling system for delayed data. This system includes a judgment module 10, a treatment day calculation module 20, and a targeted recalculation module 30. Specifically, the judgment module 10 is used to determine whether the data uploaded by the device is delayed data. The treatment day calculation module 20 is used to calculate the treatment day for the delayed data if it is delayed data, and write the treatment day into the backfilling tracking table. The treatment day is divided into time intervals from 12:00 noon of the current day to 12:00 noon of the next day. When calculating the treatment day, both UTC+8 and UTC+9 time zone calibers are used for parallel calculation. The backfilling tracking table includes three fields: device identifier, treatment day, and time zone. The targeted recalculation module 30 is used to perform targeted recalculation based on the backfilling tracking table. The targeted recalculation and regular statistics run in parallel within the same AWS Glue daily scheduling job.

[0033] In one implementation, the judgment module 10 includes: The comparison unit is used to compare the hour to which the data time belongs with the current job processing hour window; The delayed data identification unit is used to identify data as delayed data if the hour to which the data time belongs is earlier than the current job processing hour window.

[0034] In one implementation, the directional recalculation module 30 includes: The cleanup unit is used to clean up the corresponding tracking records in the backfill tracking table if the recalculation is successful. The reserved unit is used to retain the corresponding tracking record in the backfill tracking table to enable automatic retry if the recalculation fails.

[0035] The working principle of each module in the automatic backfill system for delayed data in this embodiment is the same as that of each step in the above method embodiment, and will not be repeated here.

[0036] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 3 As shown. The terminal may include one or more processors 100 ( Figure 3 (Only one is shown in the image), memory 101, and a computer program 102 stored in memory 101 that can run on one or more processors 100, such as a delayed data autofill program. When one or more processors 100 execute computer program 102, they can implement the various steps in the delayed data autofill method embodiment. Alternatively, when one or more processors 100 execute computer program 102, they can implement the functions of various modules / units in the delayed data autofill method embodiment, which is not limited here.

[0037] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0038] In one embodiment, memory 101 may be an internal storage unit of an electronic device, such as a hard drive or RAM. Memory 101 may also be an external storage device of the electronic device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, memory 101 may include both internal and external storage units. Memory 101 is used to store computer programs and other programs and data required by the terminal. Memory 101 can also be used to temporarily store data that has been output or will be output.

[0039] Those skilled in the art will understand that Figure 3 The schematic diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0040] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, operational databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual operating data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0041] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatically backfilling delayed data, characterized in that, The method includes: Determine if the data uploaded by the device is delayed; If the data is delayed, calculate the treatment date for the delayed data and write the treatment date into the backfill tracking table; Based on the backfill tracking table, perform targeted recalculation.

2. The method for automatically backfilling delayed data according to claim 1, characterized in that, The determination of whether the data uploaded by the device is delayed includes: Compare the hour to which the data time belongs with the current job processing hour window; If the hour to which the data belongs is earlier than the current job processing hour window, it is identified as delayed data.

3. The method for automatically backfilling delayed data according to claim 1, characterized in that, The targeted recalculation and regular statistics are run in parallel within the same AWS Glue daily scheduled job.

4. The method for automatically backfilling delayed data according to claim 1, characterized in that, The treatment days are divided into time intervals from 12:00 noon on the current day to 12:00 noon on the following day.

5. The method for automatically backfilling delayed data according to claim 1, characterized in that, When calculating the treatment date, parallel calculations were performed using both UTC+8 and UTC+9 time zone calibers.

6. The method for automatically backfilling delayed data according to claim 1, characterized in that, The backfill tracking form includes three fields: device identifier, treatment date, and time zone.

7. The method for automatically backfilling delayed data according to claim 1, characterized in that, The execution of targeted recalculation includes: If the recalculation is successful, the corresponding tracking records in the backfill tracking table will be cleared. If recalculation fails, the corresponding tracking record in the backfill tracking table will be retained to enable automatic retry.

8. An automatic data backfilling system for delayed data, characterized in that, The system includes: The judgment module is used to determine whether the data uploaded by the device is delayed data; The treatment day calculation module is used to calculate the treatment day for delayed data and write the treatment day into the backfill tracking table. The targeted recalculation module is used to perform targeted recalculation based on the backfill tracking table.

9. A terminal, characterized in that, The terminal includes a memory, a processor, and an automatic delay data backfilling program stored in the memory and executable on the processor. When the processor executes the automatic delay data backfilling program, it implements the steps of the automatic delay data backfilling method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a delayed data autofill program, which, when executed by a processor, implements the steps of the delayed data autofill method as described in any one of claims 1-7.

Images