Data statistics method and device and storage medium

Abstract

The invention discloses a data statistics method and device and a storage medium, and the method comprises the steps: obtaining a breakpoint record related to a statistics task, and the breakpoint record comprises a breakpoint identifier and a corresponding breakpoint time value; determining a statistical starting time based on the breakpoint record, and when the breakpoint record exists, the statistical starting time is the breakpoint time value; when the breakpoint record does not exist, the statistical starting time is the minimum update time value in the original data storage; according to the statistical starting time and the maximum updating time value in the original data storage, a date list needing statistics is determined; storing dates which do not exist in the statistical date table in the date list into the statistical date table; and updating the breakpoint time value in the breakpoint record to the maximum update time value: sequentially executing statistical operation on each date in the statistical date table, and after each statistical operation is completed, storing a statistical result into a statistical result table and deleting the corresponding date record in the statistical date table at the same time. By acquiring the breakpoint record and determining the statistical starting time, the date to be counted is automatically identified, and the trouble of manually appointing the date is avoided. And the date list is merged into the statistical date table and de-duplication processing is carried out, so that the accuracy and integrity of the statistical data are ensured.

Description

Technical Field

[0001] The present invention relates to the field of data statistics, and in particular to a data statistics method, device and storage medium. Background Art

[0002] The field of data analysis plays a crucial role in today's information age. With the continuous growth of data volumes and the increasing complexity of data application scenarios, accurate and efficient data statistics have become a key support for the development of many industries and fields. In the existing data analysis field, traditional methods are often used for data statistics. A common practice is to conduct one-time data statistics, requiring that all data for a specific date must be complete before the statistics can be performed. Once data statistics are completed, if new data for that date arrives later, the data for that date will not be counted again. Furthermore, existing statistical methods often only count each date once. If the statistics task for a certain date fails, the system will simply skip that date and move on to the next one. This makes it difficult to restart the statistics from the point where the failure occurred, which can lead to incomplete and inefficient data statistics. Furthermore, the date to which the data to be counted belongs must be manually specified, and the data cannot be automatically identified. Summary of the Invention

[0003] The purpose of the present invention is to overcome the above technical problems and provide a data statistics method, device and storage medium.

[0004] In a first aspect, the present invention provides a data statistics method comprising the following steps: Obtain a breakpoint record associated with the statistical task, wherein the breakpoint record includes a breakpoint identifier and a corresponding breakpoint time value; Determine the statistical start time based on the breakpoint record, when the breakpoint record exists, the statistical start time is the breakpoint time value; when the breakpoint record does not exist, the statistical start time is the minimum update time value in the original data storage; Determine a list of dates that require statistics based on the statistics start time and the maximum update time value in the original data storage; Storing the dates in the date list that do not exist in the statistical date table into the statistical date table; Update the breakpoint time value in the breakpoint record to the maximum update time value; Execute statistical operations on each date in the statistical date table in sequence, and after each statistical operation is completed, store the statistical results in the statistical result table and delete the corresponding date record in the statistical date table.

[0005] Preferably, the breakpoint identifier of the breakpoint record includes a time identifier or a data sequence identifier.

[0006] Preferably, the statistical operation includes constructing a statistical statement, executing the statistical statement in the original data storage to obtain a statistical result, and generating a statistical record to be stored according to the statistical result.

[0007] Preferably, the step of storing the statistical results in the statistical result table includes: Determine whether there is an existing statistical record with the same statistical date as the current statistical date in the statistical result table; When the existing statistical record exists, deleting the existing statistical record and then storing the statistical result; When the existing statistical record does not exist, the statistical result is directly stored.

[0008] Preferably, the statistical date table is used to store date records to be counted, and delete the date records after completing the statistical operation of the corresponding date.

[0009] Preferably, the update of the breakpoint time value is performed after the statistical operation is completed.

[0010] Preferably, determining the date list includes traversing all data records in the original data storage to extract dates between the statistical start time and the maximum update time value.

[0011] Preferably, after the statistical operation fails, when the method is re-executed, subsequent statistical steps are continued based on the unupdated breakpoint records.

[0012] In a second aspect, the present invention provides a computer device comprising: at least one processor; and a memory storing computer instructions executable on the processor, wherein the instructions implement the steps of the above method when executed by the processor.

[0013] In a third aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0014] In summary, the present invention includes at least one of the following beneficial technical effects: by obtaining breakpoint records and determining the statistical start time, the date to be counted is automatically identified, avoiding the trouble of manually specifying the date. The date list is merged into the statistical date table and deduplication is performed to ensure the accuracy and completeness of the statistical data. Updating the breakpoint time allows the statistical task to continue from the breakpoint, avoiding the problem of repeated statistics of data and re-statistics after the statistical task fails. The steps of performing statistical operations and updating statistical results realize the retrospective update of statistical data. When new data is added to the date that has been counted, the statistical results can be updated in time. The entire method ensures the minimization of statistical resources and improves the efficiency and accuracy of data statistics through reasonable step arrangement and data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flow chart of an embodiment of the data statistics method of the present invention; Figure 2 This is a second flow chart of the data statistics method embodiment of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the Figure 1 -Attached Figure 2 , further describing the technical solutions in the embodiments of the present invention in detail. The described embodiments are merely possible technical implementations of the present invention, but are not limited thereto. Those skilled in the art can fully combine the embodiments of the present invention, and other embodiments derived without inventive effort are also within the scope of protection of the present invention. Example

[0017] Reference Figure 1 The data statistics method provided by the embodiments of the present invention includes the steps of obtaining breakpoint records, determining the statistics start time, determining a statistical date list, merging the date list, updating the breakpoint time, performing statistical operations, and updating statistical results. Through the orderly execution of these steps, the methods achieve the following effects: automatically distinguishing the attribution date of statistical data, retroactively updating statistical data, resuming statistics at breakpoints, and minimizing statistical resources.

[0018] Specifically, the step of obtaining breakpoint records checks the breakpoint table associated with the statistics task. The breakpoint table contains a breakpoint identifier column and a breakpoint time column. The breakpoint identifier can be a time identifier or a data sequence identifier. A time identifier, typically in the form of a date or timestamp, uniquely identifies a point in time. A data sequence identifier can be a data number or ID, identifying a specific sequence of data. For example, in a sales data statistics scenario, the time identifier can be a specific sales date, and the data sequence identifier can be an order number. When checking the breakpoint table, the breakpoint ID (breakid) of the statistics task is used to query whether a breakpoint record exists for this task. If so, the breakpoint time corresponding to the breakpoint record is used as the statistics start time (starttime). If no such breakpoint record exists, the minimum update time (mintime) in the original data store is used as the statistics start time (starttime). The original data store can be a database, file system, or other storage device that stores the raw data to be counted.

[0019] After determining the statistics start time (starttime), the list of dates to be counted (stattimeList) is determined based on this start time and the maximum update time value (maxtime) in the original data store. Determining the date list (stattimeList) involves traversing all data records in the original data store and extracting the dates between the statistics start time (starttime) and the maximum update time value (maxtime). This process is similar to intercepting the period between the statistics start time and the maximum update time on a timeline, extracting the dates within that period and forming a list. For example, if the statistics start time is January 1, 2024, and the maximum update time is January 10, 2024, the date list will include all dates between January 1, 2024, and January 10, 2024.

[0020] Next, any dates in the date list (stattimeList) that aren't already in the statistical date table are stored in the statistical date table. The statistical date table is used to store date records to be counted. When a date is stored, it is checked to see if it already exists in the statistical date table. If not, it is stored, ensuring there are no duplicate dates in the statistical date table. This is like storing goods in a warehouse; new goods are only added when the warehouse already has the same goods.

[0021] After that, the value of the breakpoint time column (breaktime) corresponding to the breakpoint identifier in the breakpoint table is updated to the maximum update time value (maxtime). The purpose of this is to record the end time of this statistical task so that the next statistical task can continue from this time point.

[0022] Then enter the step of executing statistical operation. Before executing statistical operation, all dates in the date list (stattimeList) have not been completed in the statistical result table. Figure 2 Statistical operations involve constructing a statistical statement (statsql), executing the statement (statsql) in the raw data store to obtain statistical results (statresult), and generating statistical records to be stored based on the statistical results (statresult). When constructing a statistical statement (statsql), each date in the date list (stattimeList) is used to construct a specific statistical statement (statsql). This statement queries the raw data store for data for that date and performs statistical calculations. For example, for sales data statistics, a statistical statement might query information such as total sales and sales volume for a specific day. After executing the statistical statement, the corresponding statistical results are obtained, and statistical records to be stored are generated based on these results.

[0023] The last step is to update the statistical results. Figure 2 When storing statistical results (statresult) in the statistical result table, the system first checks whether there is an existing statistical record with the same statistical date as the current statistical date. If there is an existing statistical record, it will be deleted before storing the new statistical result. If there is no existing statistical record, the statistical result will be stored directly. At the same time, the corresponding date record in the statistical date table for the completed statistical data will be deleted. This step ensures that the data in the statistical result table is always up to date and that only the dates for which statistical data has not been completed are retained in the statistical date table.

[0024] The implementation principle of this embodiment is as follows: by obtaining breakpoint records and determining the statistical start time, the date to be counted is automatically identified, avoiding the trouble of manually specifying the date. The date list is merged into the statistical date table and deduplication is performed to ensure the accuracy and completeness of the statistical data. Updating the breakpoint time allows the statistical task to continue from the breakpoint, avoiding the problem of repeated statistics of data and re-statistics after the statistical task fails. The steps of executing statistical operations and updating statistical results realize the retrospective update of statistical data. When new data is added to the date that has been counted, the statistical results can be updated in time. The entire method ensures the minimization of statistical resources through reasonable step arrangement and data processing, improves the efficiency and accuracy of data statistics, and solves various problems existing in the prior art. Example

[0025] The difference between this embodiment and the above embodiment is that the storage method of breakpoint records can be varied. In addition to being stored in the breakpoint table, it can also be stored in a cache, log file, etc. For example, in some scenarios with high real-time requirements, breakpoint records can be stored in a memory cache, so that the breakpoint information can be quickly read and updated. When using a cache to store breakpoint records, a key-value pair mapping is established, with the key being the breakpoint identifier and the value being the breakpoint time. When obtaining a breakpoint record, the corresponding breakpoint time is directly searched from the cache based on the breakpoint identifier. This storage method can improve data access speed and reduce database query overhead.

[0026] The implementation principle of this embodiment is as follows: different breakpoint record storage methods can be selected based on specific application scenarios and requirements. The cache storage method is advantageous in scenarios with high real-time requirements and frequent data access, improving system response speed and performance. Storing breakpoint records in log files is suitable for scenarios requiring long-term preservation and auditing of breakpoint information. By flexibly selecting the storage method for breakpoint records, data statistics methods can be further optimized, making them more adaptable to different application environments and improving the overall performance and reliability of the system. Example

[0027] This embodiment differs from the previous embodiment in that the method for determining the date list can be optimized. In addition to traversing all data records in the original data store, indexing techniques can also be employed. For example, an index can be created for the date field in the original data store. When determining the date list, the index can be used to quickly locate the dates between the statistics start time and the maximum update time, without traversing all data records. This greatly improves the efficiency of determining the date list, especially when the data volume is very large.

[0028] The implementation principle of this embodiment is that indexing technology can speed up data search. By establishing an index on the date field, the system can quickly locate the date range that meets the requirements, reducing unnecessary data traversal operations. This not only improves the efficiency of determining the date list, but also reduces system resource consumption, allowing the data statistics method to continue to operate efficiently in a big data environment and further improving the utilization efficiency of statistical resources.

[0029] Embodiment 4: The computer device provided by the present invention includes: at least one processor; and a memory, wherein the memory stores computer instructions that can be run on the processor, and the instructions implement the steps of the above method when executed by the processor.

[0030] Embodiment 5: The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0031] The above are all preferred embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data statistics method, characterized in that: The following steps are involved: Obtain a breakpoint record associated with the statistical task, wherein the breakpoint record includes a breakpoint identifier and a corresponding breakpoint time value; Determine a statistics start time based on the breakpoint record, and when the breakpoint record exists, the statistics start time is the breakpoint time value; When the breakpoint record does not exist, the statistical start time is the minimum update time value in the original data storage; Determine a list of dates that require statistics based on the statistics start time and the maximum update time value in the original data storage; Storing the dates in the date list that do not exist in the statistical date table into the statistical date table; Update the breakpoint time value in the breakpoint record to the maximum update time value; Execute statistical operations on each date in the statistical date table in sequence, and after each statistical operation is completed, store the statistical results in the statistical result table and delete the corresponding date record in the statistical date table.

2. The data statistics method according to claim 1, characterized in that: The breakpoint identifier of the breakpoint record includes a time identifier or a data sequence identifier.

3. The data statistics method according to claim 1, characterized in that: The statistical operation includes constructing a statistical statement, executing the statistical statement in the original data storage to obtain a statistical result, and generating a statistical record to be stored according to the statistical result.

4. The data statistics method according to claim 3, characterized in that: The steps for storing statistical results in the statistical result table include: Determine whether there is an existing statistical record with the same statistical date as the current statistical date in the statistical result table; When the existing statistical record exists, deleting the existing statistical record and then storing the statistical result; When the existing statistical record does not exist, the statistical result is directly stored.

5. The data statistics method according to claim 1, characterized in that: The statistical date table is used to store date records to be counted, and delete the date records after the statistical operation of the corresponding date is completed.

6. The data statistics method according to claim 1, characterized in that: The update of the breakpoint time value is performed after the statistical operation is completed.

7. The data statistics method according to claim 1, characterized in that: Determining the date list includes traversing all data records in the original data storage to extract dates between the statistical start time and the maximum update time value.

8. The data statistics method according to claim 1, characterized in that: After the statistical operation fails, when the method is re-executed, subsequent statistical steps are continued based on the unupdated breakpoint records.

9. A computer device, characterized in that: include: at least one processor; and a memory storing computer instructions executable on the processor, wherein the instructions, when executed by the processor, implement the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images