A method, apparatus, and processing device for massive data aggregation

Abstract

This application provides a method, apparatus, and processing device for aggregating massive amounts of data, which uses elements other than time to promote more stable and efficient aggregation processing, thereby helping to ensure data security and processing efficiency. The method for aggregating massive amounts of data provided in this application includes: obtaining an initial data table currently awaiting aggregation processing, wherein different data in the initial data table are stored in a daily table format; sorting the different data in the initial data table according to time based on the timestamps indicated by the daily table format, and obtaining an ID corresponding to each data entry, where ID refers to the number obtained from the sorting; determining the start ID and end ID of the next aggregation processing step; locating the corresponding data in the initial data table according to the ID range formed by the start ID and end ID of the next aggregation processing step, and performing aggregation processing.

Description

Technical Field

[0001] This application relates to the field of data aggregation, specifically to a method, apparatus, and processing equipment for aggregating massive amounts of data. Background Technology

[0002] With the rapid development of network technology, a common situation exists for various application services: the data is not only complex and varied, but also consists of massive amounts of raw data. However, in related data analysis work, it is not necessary to analyze all the data one by one. Instead, the raw data is often aggregated and processed in a certain way through some filtering conditions to simplify and filter the data. Then, the aggregated data is further processed to generate the required reports.

[0003] The inventors of this application discovered that, although existing technologies may involve relevant filtering conditions when aggregating raw data tables in a database, the actual processing in the background defaults to time-based execution. When faced with massive data tables, if the aggregation operation is performed crudely based on time, it may lead to abnormal device restarts, network failures, and huge losses to the company or customers. In other words, the larger the amount of data aggregated in a single instance, the more system resources are consumed, and the greater the probability of device malfunction.

[0004] Clearly, when faced with massive amounts of raw data tables, existing time-based aggregation solutions may cause a series of malfunctions, affecting the normal operation of the equipment. Summary of the Invention

[0005] This application provides a method, apparatus, and processing device for aggregating massive amounts of data, which uses elements other than time to promote more stable and efficient aggregation processing, thereby helping to ensure data security and processing efficiency.

[0006] Firstly, this application provides a method for aggregating massive amounts of data, the method comprising:

[0007] Retrieve the initial data table currently awaiting aggregation processing. Different data in the initial data table are stored in the format of daily tables.

[0008] Based on the timestamps indicated by the initial data table in the daily table format, the different data in the initial data table are sorted according to time, and the ID corresponding to each data is obtained. The ID refers to the number obtained by sorting.

[0009] Determine the start ID and end ID of the next aggregation processing step;

[0010] Based on the ID range consisting of the start ID and the end ID of the next aggregation processing step, locate the corresponding data in the initial data table and perform aggregation processing.

[0011] In conjunction with the first aspect of this application, in a first possible implementation of the first aspect of this application, based on the timestamp indicated by the initial data table in the daily sub-table format, the different data in the initial data table are sorted according to time, and the ID corresponding to each data item is obtained, including:

[0012] Based on the timestamps indicated by the initial data table in the daily table format, the different data in the initial data table are sorted according to time, and the ID corresponding to each data is obtained;

[0013] The ID corresponding to each piece of data is used to form a cluster index table. The cluster index table identifies the ID corresponding to each piece of data, the maximum ID in the data of the current cluster processing stage, the original table name, the creation time of the data corresponding to the maximum ID in the data of the current cluster processing stage, and the end time of the current cluster processing stage.

[0014] Determine the start ID and end ID of the next aggregation processing stage, including:

[0015] Determine the start and end times of the next aggregation processing step;

[0016] Based on the start and end times of the next aggregation processing stage, and combined with the ID corresponding to each data entry in the aggregation index table and the end time of the current processing stage, the start ID and end ID of the next processing stage are determined.

[0017] In conjunction with the first possible implementation of the first aspect of this application, in the second possible implementation of the first aspect of this application, before determining the start ID and end ID of the next processing stage based on the start time and end time of the next aggregation processing stage, and in conjunction with the ID corresponding to each data entry in the aggregation index table and the end time of the current aggregation processing stage, the method further includes:

[0018] The amount of data in the initial data table is determined based on the time range consisting of the start and end times of the next aggregation processing stage.

[0019] If the data volume exceeds the threshold, the start ID and end ID of the next aggregation processing stage will be determined based on the start and end times of the next aggregation processing stage, combined with the ID corresponding to each data entry in the aggregation index table and the end time of the current aggregation processing stage.

[0020] If the data volume is less than the threshold, the corresponding data will be retrieved from the initial data table and aggregated within the time range defined by the start and end times of the next aggregation processing step.

[0021] In conjunction with the second possible implementation of the first aspect of this application, in the third possible implementation of the first aspect of this application, based on the start time and end time of the next aggregation processing stage, and combined with the ID corresponding to each data entry in the aggregation index table and the end time of the current aggregation processing stage, the start ID and end ID of the next aggregation processing stage are determined, including:

[0022] Based on the start and end times of the next aggregation processing stage, and combined with the ID corresponding to each data entry in the aggregation index table and the end time of the current aggregation processing stage, the initial start ID and end ID of the next aggregation processing stage are determined.

[0023] The difference between the initial start ID and the end ID of the next aggregation processing step is compared with the preset computing capacity value.

[0024] If the difference between the initial start ID and the final stop ID of the next aggregation processing step is greater than the preset computing capacity value, then the IDs between the initial start ID and the final stop ID of the next aggregation processing step are processed in batches so that the number of IDs processed in the next aggregation processing step is less than the preset computing capacity value, and the start ID and the final stop ID of the next aggregation processing step are updated.

[0025] In conjunction with the third possible implementation of the first aspect of this application, in the fourth possible implementation of the first aspect of this application, batch processing is performed with a preset calculation capacity value as the number of IDs to be processed in the next aggregation processing stage;

[0026] Alternatively, batch processing can be performed by evenly distributing the number of IDs processed in different aggregation processing stages.

[0027] In conjunction with the first possible implementation of the first aspect of this application, in the fifth possible implementation of the first aspect of this application, a converged index table is formed by the ID corresponding to each piece of data, including:

[0028] Based on the different preset aggregation processing steps, the ID corresponding to each piece of data is used to form an aggregation index table.

[0029] In conjunction with the first aspect of this application, in the sixth possible implementation of the first aspect of this application, each aggregation processing step is determined by an aggregation cycle of 1 hour.

[0030] Secondly, this application provides a massive data aggregation device, the device comprising:

[0031] The acquisition unit is used to acquire the initial data table currently waiting to be processed for aggregation. The different data in the initial data table are stored in the format of daily sub-tables.

[0032] The sorting unit is used to sort different data in the initial data table according to time based on the timestamp indicated by the initial data table in the daily table format, and to obtain the ID corresponding to each data. The ID refers to the number obtained by sorting.

[0033] The determination unit is used to determine the start ID and end ID of the next aggregation processing stage;

[0034] The aggregation unit is used to locate the corresponding data in the initial data table according to the ID range consisting of the start ID and the end ID of the next aggregation processing step, and to perform aggregation processing.

[0035] In conjunction with the second aspect of this application, in a first possible implementation of the second aspect of this application, the sorting unit is specifically used for:

[0036] Based on the timestamps indicated by the initial data table in the daily table format, the different data in the initial data table are sorted according to time, and the ID corresponding to each data is obtained;

[0037] The ID corresponding to each piece of data is used to form a cluster index table. The cluster index table identifies the ID corresponding to each piece of data, the maximum ID in the data of the current cluster processing stage, the original table name, the creation time of the data corresponding to the maximum ID in the data of the current cluster processing stage, and the end time of the current cluster processing stage.

[0038] Determine the unit, specifically for:

[0039] Determine the start and end times of the next aggregation processing step;

[0040] Based on the start and end times of the next aggregation processing stage, and combined with the ID corresponding to each data entry in the aggregation index table and the end time of the current processing stage, the start ID and end ID of the next processing stage are determined.

[0041] In conjunction with the first possible implementation of the second aspect of this application, in the second possible implementation of the second aspect of this application, the determined unit is further used for:

[0042] The amount of data in the initial data table is determined based on the time range consisting of the start and end times of the next aggregation processing stage.

[0043] If the data volume exceeds the threshold, the start ID and end ID of the next aggregation processing stage will be determined based on the start and end times of the next aggregation processing stage, combined with the ID corresponding to each data entry in the aggregation index table and the end time of the current aggregation processing stage.

[0044] If the data volume is less than the threshold, the corresponding data will be retrieved from the initial data table and aggregated within the time range defined by the start and end times of the next aggregation processing step.

[0045] In conjunction with the second possible implementation of the second aspect of this application, in the third possible implementation of the second aspect of this application, the determined unit is specifically used for:

[0046] Based on the start and end times of the next aggregation processing stage, and combined with the ID corresponding to each data entry in the aggregation index table and the end time of the current aggregation processing stage, the initial start ID and end ID of the next aggregation processing stage are determined.

[0047] The difference between the initial start ID and the end ID of the next aggregation processing step is compared with the preset computing capacity value.

[0048] If the difference between the initial start ID and the final stop ID of the next aggregation processing step is greater than the preset computing capacity value, then the IDs between the initial start ID and the final stop ID of the next aggregation processing step are processed in batches so that the number of IDs processed in the next aggregation processing step is less than the preset computing capacity value, and the start ID and the final stop ID of the next aggregation processing step are updated.

[0049] In conjunction with the third possible implementation of the second aspect of this application, in the fourth possible implementation of the second aspect of this application, batch processing is performed using a preset calculation capacity value as the number of IDs to be processed in the next aggregation processing stage;

[0050] Alternatively, batch processing can be performed by evenly distributing the number of IDs processed in different aggregation processing stages.

[0051] In conjunction with the first possible implementation of the second aspect of this application, in the fifth possible implementation of the second aspect of this application, the sorting unit is specifically used for:

[0052] Based on the different preset aggregation processing steps, the ID corresponding to each piece of data is used to form an aggregation index table.

[0053] In conjunction with the second aspect of this application, in the sixth possible implementation of the second aspect of this application, each aggregation processing step is determined by an aggregation cycle of 1 hour.

[0054] Thirdly, this application provides a processing device, including a processor and a memory, wherein a computer program is stored in the memory, and when the processor invokes the computer program in the memory, it executes the method provided by the first aspect of this application or any possible implementation of the first aspect of this application.

[0055] Fourthly, this application provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to perform the method provided in the first aspect of this application or any possible implementation thereof.

[0056] From the above, it can be concluded that this application has the following beneficial effects:

[0057] To address the need for aggregating massive amounts of data, this application, after obtaining the initial data table awaiting aggregation processing, sorts the different data in the initial data table according to time based on the timestamps indicated in the daily table format, obtaining the ID corresponding to each data entry. After determining the start ID and end ID of the next aggregation processing step, it locates the corresponding data in the initial data table according to the ID range formed by the start and end IDs of the next aggregation processing step, and then performs the aggregation processing. In this process, this application uses IDs to replace the time element in existing technologies, thereby accurately locating different data in the initial data table based on IDs, providing greater controllability. Using elements other than time promotes more stable and efficient aggregation processing, thus helping to ensure data security and processing efficiency. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a flowchart illustrating the method for aggregating massive amounts of data in this application;

[0060] Figure 2 A logical diagram illustrating the generation of a converged index table for this application;

[0061] Figure 3 This is a logical diagram illustrating the aggregation of data in this application;

[0062] Figure 4 This is a schematic diagram of a structure for a massive data aggregation device according to this application.

[0063] Figure 5This is a schematic diagram of one type of processing equipment used in this application. Detailed Implementation

[0064] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0065] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved.

[0066] The module division described in this application is a logical division. In practical applications, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection between modules shown or discussed may be through some interfaces, and the indirect coupling or communication connection between modules may be electrical or other similar forms, none of which are limited in this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules. Some or all of the modules may be selected to achieve the purpose of the solution in this application according to actual needs.

[0067] Before introducing the method for aggregating massive amounts of data provided in this application, we will first introduce the background information involved in this application.

[0068] The massive data aggregation method, apparatus, and computer-readable storage medium provided in this application can be applied to processing devices to promote more stable and efficient aggregation processing by incorporating elements other than time, thereby helping to ensure data security and processing efficiency.

[0069] The massive data aggregation method mentioned in this application can be implemented by a massive data aggregation device, or by different types of processing devices such as servers, physical hosts, or even user equipment (UE) that integrate the massive data aggregation device. The massive data aggregation device can be implemented in hardware or software. The UE can specifically be a terminal device such as a smartphone, tablet, laptop, desktop computer, or personal digital assistant (PDA). The processing devices can be configured in a device cluster.

[0070] It is understood that the massive data aggregation method provided in this application is mainly for network services involving large-scale data processing tasks, such as network security services. Therefore, the processing equipment is usually a server. Of course, in flexible practical applications, it can also be configured on various processing equipment in different application scenarios according to actual needs, so as to obtain a more stable and efficient data aggregation target.

[0071] The following section introduces the method for aggregating massive amounts of data provided in this application.

[0072] First, refer to Figure 1 , Figure 1 This paper illustrates a flowchart of a method for aggregating massive data according to this application. The method for aggregating massive data provided by this application may specifically include the following steps S101 to S104:

[0073] Step S101: Obtain the initial data table currently waiting to be aggregated. The different data in the initial data table are stored in the format of daily tables.

[0074] It is understandable that for aggregation processing, this application deals with a data table, which can also be called a data table. The data table obtained at the beginning is referred to as the initial data table in this application.

[0075] In this step S101, the acquisition of the initial data table is relatively flexible in terms of the source and method of acquisition, and can refer to the existing technology. The difference between this application and the existing technology is that after the data table is acquired and before the aggregation process is executed, data processing is inserted (i.e., the following steps S102 to S104) to ensure the stability and processing efficiency of the aggregation process.

[0076] The initial data table involved in this application is stored in a daily table format, which records the specific generation time of each data item in the table by recording a timestamp. Daily table is a commonly used table storage mechanism. Existing technology can perform aggregation processing based on time based on this mechanism. The specific content of the daily table mechanism will not be elaborated in this application.

[0077] Step S102: Based on the timestamps indicated by the initial data table in the daily table format, sort the different data in the initial data table according to time and obtain the ID corresponding to each data item. The ID refers to the number obtained by sorting.

[0078] As can be seen, this application introduces an ID mechanism, which uses ID to indicate the number obtained by sorting each piece of data in the initial data table according to the original record time. In this way, each piece of data is evenly identified by ID, and there will be no duplicate IDs.

[0079] Specifically, please refer to Table 1 below, which records the processing results of a certain instance data table:

[0080] Table 1 - Processing Results of the Example Data Table

[0081]

[0082]

[0083] The table name "webaccess_20230301" refers to the original data table storing web access logs on March 1, 2023. The format of this table name is only an example and has no practical significance in the scheme of this application.

[0084] For example, the initial data table name format can be table_name_yyyymmdd, where yyyymmdd represents the date format of year, month, and day, such as raw_table_2023025 or gather_table_2023026, etc.

[0085] The specific format of the table name can involve both the application specifications of the specific database and the operating specifications of the staff. For example, the data tables involved in this application can involve relational databases such as PostgreSQL and MySQL.

[0086] Step S103: Determine the start ID and end ID of the next aggregation processing step;

[0087] After identifying each piece of data in the initial data table with the ID configured in this application, this application can extract the corresponding data from the initial data table based on the determined ID range used to perform the aggregation process, so as to perform the aggregation process.

[0088] For the ID range used to perform the next aggregation process (i.e., the next aggregation process step), from the perspective of convenient operation, it can be determined by the start ID and the end ID. Since each piece of data in the initial data table is a unique ID obtained by sorting, the ID range can be quickly and accurately determined by the start ID and the end ID.

[0089] It should be understood that step S103 in this application does not only involve determining the start ID and end ID once. After obtaining the ID of each data in the initial data table, since the objects of each aggregation process are executed through the ID range, there may be situations where multiple aggregation processes are required. Therefore, step S103 after step S102 may involve multiple processes, that is, it may involve determining the start ID and end ID of the next aggregation process multiple times, and the aggregation process can be updated.

[0090] Furthermore, it should not be ruled out that step S103 can also be independent of step S102. That is, it is not necessary to rely on the ID determined in step S102 to determine the ID range of the next aggregation processing stage after step S102 has been executed. For example, the ID range of each aggregation processing stage can be fixed directly, or the ID range of each aggregation processing stage can be set randomly. The specific determination scheme of the ID range can be flexibly set according to actual needs. If a specific ID needs to be referenced in the process of determining the ID range, it obviously needs to rely on the ID determined in step S102.

[0091] Step S104: Locate the corresponding data in the initial data table according to the ID range consisting of the start ID and the end ID of the next aggregation processing step, and perform aggregation processing.

[0092] Once the range of IDs consisting of the starting ID and the ending ID for the next aggregation processing stage is determined, data can be retrieved from the initial data table to perform aggregation processing based on this data.

[0093] Specifically, one can either start from the starting ID, process each item in the initial data table one by one, and perform aggregation processing on each item until the ending ID; or one can first process the data from the starting ID to the ending ID, and then perform aggregation processing in a centralized manner.

[0094] Clearly, guided by the ID range consisting of the start ID and the end ID of the next aggregation processing stage, locating and retrieving data from the initial data table to perform aggregation processing is an adaptive setting based on the actual aggregation processing mechanism.

[0095] Furthermore, it is understood that this application does not make specific improvements to the aggregation process, but rather to the mechanism for determining the processing objects used to perform the aggregation process. That is, preprocessing (which can also be understood as secondary processing) is introduced before the aggregation process. The specific data aggregation strategy adopted by the aggregation process to simplify and filter the data can be flexibly adjusted according to the actual situation, and this application does not impose specific limitations on it.

[0096] In the process of initiating aggregation processing based on ID, its superiority can be highlighted by comparing it with existing technologies.

[0097] Specifically, existing technologies initiate aggregation processing by directly fetching data based on time. For example, the time range is determined based on the start time (start_time) and end time (end_time) to fetch data, and create_time >= start_time and create_time <= end_time are used as filtering conditions in SQL statements.

[0098] However, taking the table name "20230301" as an example, SQL statements like `select count(*) from raw_table_name_20230301 where create_time>=start_time and create_time<=end_time` are very unsafe. When dealing with large amounts of data, especially when there is high-density data within a defined time frame, it can lead to excessive consumption of system resources. If the hardware and software conditions of the device are insufficient, it may cause the device to restart abnormally, resulting in network failures, and may even lead to data corruption, loss, and other data security incidents.

[0099] This application proposes to initiate aggregation processing by ID instead of by time. By using a specific ID, the scope / object of ID processing can be accurately determined, thereby significantly improving the controllability of the initiated aggregation processing. This is unlike existing technologies that use time to determine the processing scope, which face the problem of very large amounts of unknown data.

[0100] In summary, in response to the need for massive data aggregation, this application, after obtaining the initial data table awaiting aggregation processing, sorts the different data in the initial data table according to time based on the timestamps indicated in the daily table format, and obtains the ID corresponding to each data item. After determining the start ID and end ID of the next aggregation processing step, the corresponding data in the initial data table is located according to the ID range formed by the start ID and end ID of the next aggregation processing step, and the aggregation processing is performed. In this process, this application uses IDs to replace the time element of the prior art, thereby accurately locating different data in the initial data table based on IDs, which is more controllable. Using elements other than time promotes more stable and efficient aggregation processing, thereby helping to ensure data security and processing efficiency.

[0101] Continue with the above Figure 1 The steps of the illustrated embodiment and their possible implementation methods in practical applications are described in detail.

[0102] To facilitate implementation and improve processing efficiency, this application also introduces the concept of an index as an exemplary implementation method for the aforementioned data processing between obtaining the initial data table and initiating the aggregation process. An aggregation index table is configured to serve the pre-processing involved, which may involve the preceding ID conversion and ID range determination.

[0103] Specifically, step S102, which sorts different data in the initial data table according to time based on the timestamps indicated by the initial data table in daily sub-table format, and obtains the ID corresponding to each data entry, may include:

[0104] Based on the timestamps indicated by the initial data table in the daily table format, the different data in the initial data table are sorted according to time, and the ID corresponding to each data is obtained;

[0105] The ID corresponding to each piece of data is used to form a cluster index table. The cluster index table identifies the ID corresponding to each piece of data, the maximum ID in the data of the current cluster processing stage, the original table name, the creation time of the data corresponding to the maximum ID in the data of the current cluster processing stage, and the end time of the current cluster processing stage.

[0106] This aggregated index table is formed as a result of step S102, making it convenient for subsequent related data processing to directly call upon it.

[0107] In conjunction with the data table "webaccess_20230301" mentioned earlier, you can also refer to Table 2 below for a more intuitive understanding of the clustered index table here.

[0108] Table 2 - Example of a clustered index table

[0109]

[0110]

[0111] Table 2 records the cluster index information of Table 1. For simplicity, the cluster index table only lists the information of one original data table.

[0112] As can be seen from Table 2, the maximum ID in the data of the current aggregation processing stage can be indicated by table_id, the original table name can be indicated by table_name, the creation time of the data corresponding to the maximum ID in the current aggregation processing stage can be indicated by id_create_time, and the end time of the current aggregation processing stage can be indicated by create_time.

[0113] Furthermore, Table 2 contains repeated and consecutive table_id-6003, indicating that these rows of data all correspond to the same aggregation processing stage, and ID-6003 is the maximum ID in that aggregation processing stage.

[0114] With the introduction of a convergence index table setting here, step S103, in determining the start ID and end ID of the next convergence processing stage, can include:

[0115] Determine the start and end times of the next aggregation processing step;

[0116] Based on the start and end times of the next aggregation processing stage, and combined with the ID corresponding to each data entry in the aggregation index table and the end time of the current processing stage, the start ID and end ID of the next processing stage are determined.

[0117] As can be seen, in this embodiment, the processing range can also be determined by time. However, unlike the prior art, this application can combine the introduced aggregation index table and use the time in the data as a guide to convert between time range and ID range. This allows for a smooth integration into application scenarios that originally initiated aggregation processing based on time range, making it easier to improve the practicality and applicability of this application in actual applications.

[0118] Specifically, existing technologies initiate aggregation processing based on the start time (start_time) and end time (end_time), directly using create_time >= start_time and create_time <= end_time as the filtering conditions for the SQL statement.

[0119] This application converts the start time (start_time) into the start ID (min_id) and the end time (end_time) into the end ID (maximum id), and uses id>min_id and id<=max_id as the filtering condition for the SQL statement.

[0120] Furthermore, as mentioned above, this application aims to address the shortcomings of existing technologies that result in unstable processing effects when using time-based aggregation processing to handle massive amounts of data. However, with small amounts of data, the processing effects of both methods may appear to be the same, as it is difficult to expose existing technical defects when processing small amounts of data.

[0121] Therefore, in this situation, to improve operational convenience and simplify the process at a more nuanced level, the traditional time-based aggregation processing setting can still be used. Specifically, as another practical implementation, before determining the start ID and end ID of the next processing stage based on the start and end times of the next aggregation processing stage in the previous embodiment, combined with the ID corresponding to each data entry in the aggregation index table and the end time of the current aggregation processing stage, the method of this application may further include:

[0122] The amount of data in the initial data table is determined based on the time range consisting of the start and end times of the next aggregation processing stage.

[0123] If the data volume exceeds the threshold, the start ID and end ID of the next aggregation processing stage will be determined based on the start and end times of the next aggregation processing stage, combined with the ID corresponding to each data entry in the aggregation index table and the end time of the current aggregation processing stage.

[0124] If the data volume is less than the threshold, the corresponding data will be retrieved from the initial data table and aggregated within the time range defined by the start and end times of the next aggregation processing step.

[0125] The threshold for the amount of data involved here can be configured / adjusted according to the operator's experience, or it can be determined automatically by the equipment when there is no significant difference in the processing effect of the two schemes (and can be adjusted automatically in real time).

[0126] Furthermore, as mentioned earlier, the use of IDs to initiate aggregation processing in this application provides greater controllability and thus better stability in processing performance. In this case, for the ID range processing based on the aggregation index table in the previous embodiments, this application can further introduce precise and controllable processing quantity settings to further enhance controllability.

[0127] Specifically, in the preceding embodiments, based on the start and end times of the next aggregation processing stage, and combining the ID corresponding to each data entry in the aggregation index table with the end time of the current aggregation processing stage, the process of determining the start ID and end ID of the next aggregation processing stage may include the following:

[0128] Based on the start and end times of the next aggregation processing stage, and combined with the ID corresponding to each data entry in the aggregation index table and the end time of the current aggregation processing stage, the initial start ID and end ID of the next aggregation processing stage are determined.

[0129] The difference between the initial start ID and the end ID of the next aggregation processing step is compared with the preset computing capacity value.

[0130] If the difference between the initial start ID and the final stop ID of the next aggregation processing step is greater than the preset computing capacity value, then the IDs between the initial start ID and the final stop ID of the next aggregation processing step are processed in batches so that the number of IDs processed in the next aggregation processing step is less than the preset computing capacity value, and the start ID and the final stop ID of the next aggregation processing step are updated.

[0131] As can be seen, here the ID range involved in the initially determined next aggregation processing step is divided according to the preset computing capacity value. If it exceeds the computing capacity value, it needs to be divided. The IDs between the initial ID range, i.e. the initial start ID and the end ID, are processed in batches. This reduces the computing load of one aggregation processing step, reduces the burden on the equipment, promotes higher stability, and ensures stable and efficient processing.

[0132] Let the calculated carrying capacity value (limit carrying capacity value) be denoted as limit_value. If the termination ID max_id - the starting ID min_id > the calculated carrying capacity value limit_value, then all the data cannot be aggregated at once. Instead, it should be aggregated in batches. Assuming the calculated carrying capacity value is 100,000 data entries, and the termination ID max_id - the starting ID min_id = 1 million, it is safe and reasonable to aggregate in 10 batches to ensure stability.

[0133] It is worth adding that, in existing technologies that directly rely on time to initiate aggregation processing, for example, if 1 million data points need to be aggregated, it is difficult to reasonably divide the data range for extreme aggregation by dividing the time period into equal segments, because the generation of data is not evenly distributed over time. Data will be concentrated in a certain period of time, while there will be almost no data in another period of time, and this situation is very common. However, when ID is introduced to initiate aggregation processing, this application continues to use batch processing based on the calculated carrying capacity value, which can ensure safe and accurate aggregation and effectively control the use of equipment resources.

[0134] Among them, as another practical implementation method, the batch processing mentioned here can be executed with a preset calculation capacity value as the number of IDs processed in the next aggregation processing stage. This setting is very convenient to implement and is more conducive to ensuring efficient data processing.

[0135] Alternatively, batch processing can be implemented by evenly distributing the number of IDs processed in different aggregation processing stages. This setting can balance the processing volume between different aggregation processing stages, thereby making the overall data processing more stable.

[0136] It should be understood that the batch processing involved in this application can be implemented not only by batching according to quantity or by batching evenly, but also by other batching strategies, which can be configured according to the actual situation.

[0137] In addition, for the aggregation processing stage, besides the flexibility that can be adjusted in the previous embodiments, it can also be pre-set. Of course, for the aggregation processing stage in this case, factors that may affect the processing stability, such as the amount of data processed, can be fully considered in advance to make the determination of the effect of the aggregation processing stage more intelligent.

[0138] For example, the ID range for each round of aggregation processing can be determined in advance. This can be either determining the ID span, such as an ID range with a length of 100, or directly determining the specific start ID and end ID.

[0139] In this case, for the aggregation index table of service aggregation processing, the process of forming the aggregation index table with the ID corresponding to each piece of data in the previous embodiment can include:

[0140] Based on the different preset aggregation processing steps, the ID corresponding to each piece of data is used to form an aggregation index table.

[0141] As can be seen, with the aggregation processing steps predetermined, they can be used as a guide to generate a more accurate aggregation index table, thereby enabling the aggregation index table to play a more accurate reference role for the related aggregation processing involved later.

[0142] Furthermore, for each aggregation processing step involved in this application (including the aggregation processing step initially determined in the preceding step switching mechanism), as another exemplary implementation method, it is specifically determined by a 1-hour aggregation cycle. It can be understood that this approach is more inclined to automated execution on the server, thereby automatically initiating the corresponding aggregation processing every 1 hour throughout the day, and automatically providing data basis for relevant application services on the server.

[0143] The following examples will be used to illustrate the exemplary embodiments mentioned above in a more vivid way.

[0144] I. Preliminary Preparations

[0145] Prerequisites required for implementing this invention.

[0146] 1. The databases related to network devices use relational databases such as PostgreSQL and MySQL.

[0147] 2. The database tables are in the form of daily tables. This application describes the data in the form of daily tables. See Table 1 above. webaccess_20230301 represents the original data table storing web access logs on March 1, 2023.

[0148] 3. The limit value for data aggregation, assuming it is 100,000, can be configured as needed to suit low, medium and high-end products.

[0149] 4. The aggregation period T is assumed to be 1 hour. As time progresses, the raw data on the device continuously increases. Each aggregation only aggregates data that has not yet been aggregated within the most recent period; already aggregated data is not aggregated again. Periodic aggregation reduces the load on the device; aggregating data at regular intervals fully utilizes device resources and quickly captures the latest data changes.

[0150] 5. The clustered index table still uses a daily partitioned table format. The clustered index table contains five fields: id, table_id, table_name, id_create_time, and create_time. The index table for each day only records the maximum id value of the original data within each time period. In addition to its own id value, the clustered index table also needs to record the maximum table_id value of each original table within each aggregation time period, the table name of the original table (table_name), the original data generation time (id_create_time) corresponding to the maximum table_id in the current time period, and the end time (create_time) of the current aggregation.

[0151] 6. Explanation of the meaning of each field in the clustered index table.

[0152] Referring to Table 2 above, id represents the ID number of the data recorded in the aggregated index table, and id=1 indicates the first record; table_name=webaccess_20230301 indicates that this data is an index record of the original table webaccess_20230301; create_time='2023-02-24 01:00:00' indicates the end time of this aggregation; 557 indicates the maximum table_id value within this time period, that is, the maximum id value before 01:00:00 on 2023-03-01; the original data generation time corresponding to the maximum id is recorded by id_create_time, id_create_time='2023-03-01 00:59:54'. In summary, before 01:00:00 on 2023-03-01, the last record in the original data table webaccess_20230301 was dated 00:59:54 on 2023-03-01, with a corresponding table_id of 557.

[0153] II. Generation of the Cluster Index Table

[0154] refer to Figure 2 The diagram shown illustrates a logical approach to generating a converged index table according to this application. The generation of the converged index table in this application includes:

[0155] 1. Upon initial power-up of the network device, the aggregation index table is initialized, and the aggregation flag parameter `do_optmize` is initialized to 0. A value of 1 for `do_optmize` indicates that the aggregation index table needs to be updated, while a value of 0 indicates that no update is needed or that the update has already been completed and does not need to be repeated. The next aggregation timestamp `next_gather_time` is initialized to 0.

[0156] 2. Calculate the current timestamp current_time.

[0157] 3. If next_gather_time == 0, it indicates initial power-on. Set do_optmize to 1 and proceed to step 5.

[0158] 4. If current_time >= next_gather_time, it means the next gathering cycle has arrived, and do_optmize is assigned the value 1, 5.

[0159] 5. If do_optmize == 0, go to 2 and recalculate the new current time; if do_optmize == 1, go to 6 and update next_gather_time.

[0160] 6. Update next_gather_time. The calculation method for next_gather_time is that next_gather_time equals the current time plus the time remaining until the next calculation cycle.

[0161] For example, assuming the current time is 2023-02-25 15:37:15, the next convergence point will be after 2023-02-25 16:00:00, and the remaining time to reach the next cycle is 23 minutes and 45 seconds. Assuming the current time is h hours, m branches, and s seconds, and disregarding the year, month, and day, the formula for calculating the convergence time of the next cycle is as follows:

[0162] next_gather_time=h+(60-m)*60+(60-s).

[0163] Where h is the number of hours corresponding to the current timestamp current_time, m is the number of minutes, and s is the number of seconds of the current time.

[0164] 7. Record the max_id value of the original data table within each aggregation time period.

[0165] select max(id)from webaccess_20230301where create_time<'2023-02-2401:00:00'.

[0166] 8. Query the original data table generation time based on max_id found in step 7.

[0167] select id,create_time as id_create_time from webaccess_20230301whereid=557.

[0168] The accuracy of the calculated max_id can be verified by using the reverse lookup method.

[0169] 9. The query results for max_id and the corresponding data generation time of the maximum id are inserted into the clustered index table using the INSERT INTO statement. See the first row of data in Table 2 above.

[0170] 10. Repeating steps 7 through 9 will record the maximum id value information of all original data tables into the clustered index table.

[0171] 11. Repeating steps 2 to 10 allows for continuous recording of information within each aggregation time period and storing it in the aggregation index table.

[0172] III. Aggregation of Original Data Tables

[0173] refer to Figure 3 The diagram shown illustrates a logical representation of the data aggregation in this application. The aggregation processing of data tables in this application includes:

[0174] 1. The key point of this invention is to convert the start time and end time of the aggregation into the minimum ID and maximum ID required for aggregation, respectively, and then perform aggregation based on the ID value.

[0175] 2. First, calculate the minimum and maximum ID values ​​of the original data table. Let table_min_id represent the minimum ID of the original data, and table_max_id represent the maximum ID of the original data table.

[0176] The statement `select min(id) from table;` retrieves the value of `table_min_id`. Similarly, the statement `select max(id) from table;` retrieves the value of `table_max_id`.

[0177] If table_min_id <= 0, it means the table has no data, no aggregation is needed, and no further operations will be performed.

[0178] 3. Calculate the maximum ID value within the aggregation period, i.e., convert end_time to max_id.

[0179] (1) If the current aggregation period is from 23:00 to 24:00, max_id = table_max_id.

[0180] (2) If the current aggregation period is any hour between 0:00 and 23:00, use the following SQL query:

[0181] select min(table_id)as max_id from gather_table_yyyymmdd where table_name='raw_table_yyyymmdd'and create_time>=next_start_time and create_time <next_end_time。

[0182] Wherein, next_start_time represents the start time of the next aggregation cycle, and next_end_time represents the end time of the next aggregation cycle of the current aggregation cycle.

[0183] next_start_time - end_time = 1 (seconds), the two differ by 1 second.

[0184] If max_id <= 0, it means there is no data or the index table has not yet been generated, and no further operations will be performed.

[0185] 4. Calculate the minimum ID value within the aggregation period, i.e., convert start_time to min_id.

[0186] (1) If the current aggregation period is from 0 to 1, min_id = table_min_id.

[0187] (2) If the current collection period is any hour between 1:00 and 23:00, use the following SQL query:

[0188] select min(table_id)as min_id from gather_table_yyyymmdd where table_name='raw_table_yyyymmdd'and create_time>=start_time and create_time <end_time。

[0189] If the min_id is less than or equal to 0, continue searching for the maximum table_id value before start_time. The SQL statement is as follows:

[0190] select max(table_id)+1as min_id from gather_table_yyyymmdd wherecreate_time <start_time and table_name=‘raw_table_yyyymmdd’。

[0191] 5. Determine the batch to be aggregated.

[0192] Compare max_id - min_id with the database's maximum capacity value, limit_value.

[0193] If max_id - min_id <= limit_value, then the SQL query condition is: where id > min_id and id <= max_id and other_conditions.

[0194] If max_id - min_id > limit_value, then it needs to be done in batches, with a maximum of limit_value data entries aggregated each time. Alternatively, it can be aggregated evenly according to the actual situation. For example, if max_id - min_id = 120,000 and limit_value = 100,000, 60,000 entries can be aggregated each time, for a total of 2 batches.

[0195] 6. Store the aggregated results in a new table for later processing.

[0196] 7. If there are N tables that need to be merged, then repeat steps 1-6 until the merging is complete.

[0197] The above is an introduction to the massive data aggregation method provided in this application. In order to facilitate better implementation of the massive data aggregation method provided in this application, this application also provides a massive data aggregation device from the perspective of functional modules.

[0198] See Figure 4 , Figure 4 This is a schematic diagram of a structure for a massive data aggregation device according to this application. In this application, the massive data aggregation device 400 may specifically include the following structure:

[0199] The acquisition unit 401 is used to acquire the initial data table currently waiting to be processed for aggregation. Different data in the initial data table are stored in the format of daily tables.

[0200] The sorting unit 402 is used to sort different data in the initial data table according to time based on the timestamp indicated by the initial data table in the daily table format, and obtain the ID corresponding to each data. The ID refers to the number obtained by sorting.

[0201] The determining unit 403 is used to determine the start ID and end ID of the next aggregation processing step;

[0202] The aggregation unit 404 is used to locate the corresponding data in the initial data table according to the ID range consisting of the start ID and the end ID of the next aggregation processing step, and to perform aggregation processing.

[0203] In conjunction with the second aspect of this application, in a first possible implementation of the second aspect of this application, the sorting unit 402 is specifically used for:

[0204] Based on the timestamps indicated by the initial data table in the daily table format, the different data in the initial data table are sorted according to time, and the ID corresponding to each data is obtained;

[0205] The ID corresponding to each piece of data is used to form a cluster index table. The cluster index table identifies the ID corresponding to each piece of data, the maximum ID in the data of the current cluster processing stage, the original table name, the creation time of the data corresponding to the maximum ID in the data of the current cluster processing stage, and the end time of the current cluster processing stage.

[0206] Unit 403 is specifically used for:

[0207] Determine the start and end times of the next aggregation processing step;

[0208] Based on the start and end times of the next aggregation processing stage, and combined with the ID corresponding to each data entry in the aggregation index table and the end time of the current processing stage, the start ID and end ID of the next processing stage are determined.

[0209] In one exemplary implementation, the determining unit 403 is further configured to:

[0210] The amount of data in the initial data table is determined based on the time range consisting of the start and end times of the next aggregation processing stage.

[0211] If the data volume exceeds the threshold, the start ID and end ID of the next aggregation processing stage will be determined based on the start and end times of the next aggregation processing stage, combined with the ID corresponding to each data entry in the aggregation index table and the end time of the current aggregation processing stage.

[0212] If the data volume is less than the threshold, the corresponding data will be retrieved from the initial data table and aggregated within the time range defined by the start and end times of the next aggregation processing step.

[0213] In yet another exemplary implementation, the determining unit 403 is specifically used for:

[0214] Based on the start and end times of the next aggregation processing stage, and combined with the ID corresponding to each data entry in the aggregation index table and the end time of the current aggregation processing stage, the initial start ID and end ID of the next aggregation processing stage are determined.

[0215] The difference between the initial start ID and the end ID of the next aggregation processing step is compared with the preset computing capacity value.

[0216] If the difference between the initial start ID and the final stop ID of the next aggregation processing step is greater than the preset computing capacity value, then the IDs between the initial start ID and the final stop ID of the next aggregation processing step are processed in batches so that the number of IDs processed in the next aggregation processing step is less than the preset computing capacity value, and the start ID and the final stop ID of the next aggregation processing step are updated.

[0217] In another exemplary implementation, batch processing is performed with a preset computing capacity value as the number of IDs to be processed in the next aggregation processing stage;

[0218] Alternatively, batch processing can be performed by evenly distributing the number of IDs processed in different aggregation processing stages.

[0219] In yet another exemplary implementation, the sorting unit 402 is specifically used for:

[0220] Based on the different preset aggregation processing steps, the ID corresponding to each piece of data is used to form an aggregation index table.

[0221] In yet another exemplary implementation, each aggregation processing step is determined by a 1-hour aggregation cycle.

[0222] This application also provides a processing device from a hardware architecture perspective, see [link / reference]. Figure 5 , Figure 5 This diagram illustrates a structural schematic of the processing device of this application. Specifically, the processing device may include a processor 501, a memory 502, and an input / output device 503. The processor 501 executes the computer program stored in the memory 502 to implement, for example... Figure 1 The steps of the massive data aggregation method in the corresponding embodiment; or, when the processor 501 executes the computer program stored in the memory 502, it implements as follows: Figure 4 Corresponding to the functions of each unit in the embodiment, the memory 502 is used to store the functions executed by the processor 501 as described above. Figure 1 The computer program required for the massive data aggregation method in the corresponding embodiment.

[0223] For example, a computer program may be divided into one or more modules / units, one or more of which are stored in memory 502 and executed by processor 501 to complete this application. One or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a computer device.

[0224] The processing device may include, but is not limited to, processor 501, memory 502, and input / output device 503. Those skilled in the art will understand that the illustrations are merely examples of the processing device and do not constitute a limitation on the processing device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the processing device may also include network access devices, buses, etc., and processor 501, memory 502, input / output device 503, etc., are connected via a bus.

[0225] Processor 501 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the processing device, connecting various parts of the device through various interfaces and lines.

[0226] The memory 502 can be used to store computer programs and / or modules. The processor 501 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 502 and by calling data stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function, etc.; the data storage area may store data created according to the use of the processing device, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0227] When processor 501 executes a computer program stored in memory 502, it can specifically perform the following functions:

[0228] Retrieve the initial data table currently awaiting aggregation processing. Different data in the initial data table are stored in the format of daily tables.

[0229] Based on the timestamps indicated by the initial data table in the daily table format, the different data in the initial data table are sorted according to time, and the ID corresponding to each data is obtained. The ID refers to the number obtained by sorting.

[0230] Determine the start ID and end ID of the next aggregation processing step;

[0231] Based on the ID range consisting of the start ID and the end ID of the next aggregation processing step, locate the corresponding data in the initial data table and perform aggregation processing.

[0232] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the massive data aggregation device, processing equipment, and their corresponding units described above can be found in, for example... Figure 1 The specific details of the data aggregation method in the corresponding embodiment will not be repeated here.

[0233] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0234] Therefore, this application provides a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute the present application. Figure 1 For details on the steps of the massive data aggregation method in the corresponding embodiment, please refer to the following: Figure 1 The description of the massive data aggregation method in the corresponding embodiment will not be repeated here.

[0235] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0236] Because of the instructions stored in the computer-readable storage medium, the present application can be executed as described above. Figure 1 The steps of the massive data aggregation method in the corresponding embodiment can therefore achieve the results of this application. Figure 1 The beneficial effects that the massive data aggregation method in the corresponding embodiment can achieve are detailed in the preceding description and will not be repeated here.

[0237] The above provides a detailed description of the method, apparatus, processing device, and computer-readable storage medium for aggregating massive amounts of data provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for aggregating massive amounts of data, characterized in that, The method includes: Obtain the initial data table currently awaiting aggregation processing, wherein different data in the initial data table are stored in the format of daily tables; Based on the timestamps indicated by the initial data table in the daily table format, the different data in the initial data table are sorted according to time, and the ID corresponding to each data is obtained, where the ID refers to the number obtained by sorting. Determine the start ID and end ID of the next aggregation processing step; According to the ID range consisting of the start ID and the end ID of the next aggregation processing step, locate the corresponding data in the initial data table and perform the aggregation processing; Based on the timestamps indicated by the initial data table in the daily table format, the different data in the initial data table are sorted by time, and an ID corresponding to each data entry is obtained, including: Based on the timestamp indicated by the initial data table in the daily table format, the different data in the initial data table are sorted according to time, and the ID corresponding to each data is obtained; The ID corresponding to each piece of data is used to form a convergence index table, wherein the convergence index table identifies the ID corresponding to each piece of data, the maximum ID in the data of the current convergence processing stage, the original table name, the creation time of the data corresponding to the maximum ID in the data of the current convergence processing stage, and the end time of the current convergence processing stage; Determining the start ID and end ID of the next aggregation processing step includes: Determine the start and end times of the next aggregation processing step; Based on the start and end times of the next aggregation processing stage, and combined with the ID corresponding to each data entry in the aggregation index table and the end time of the current aggregation processing stage, the start ID and end ID of the next aggregation processing stage are determined.

2. The method according to claim 1, characterized in that, Before determining the start ID and end ID of the next aggregation processing stage based on the start and end times of the next aggregation processing stage, combined with the ID corresponding to each data entry in the aggregation index table and the end time of the current aggregation processing stage, the method further includes: Based on the time range consisting of the start time and the end time of the next aggregation processing stage, the amount of data in the initial data table is determined. If the amount of data exceeds the threshold, then based on the start and end times of the next aggregation processing stage, and combined with the ID corresponding to each data entry in the aggregation index table and the end time of the current aggregation processing stage, the start ID and end ID of the next aggregation processing stage are determined. If the amount of data is less than the threshold, the corresponding data will be retrieved from the initial data table and the aggregation process will be executed within the time range defined by the start and end times of the next aggregation process.

3. The method according to claim 2, characterized in that, Based on the start and end times of the next aggregation processing stage, and in conjunction with the ID corresponding to each data entry in the aggregation index table and the end time of the current aggregation processing stage, the start ID and end ID of the next aggregation processing stage are determined, including: Based on the start and end times of the next aggregation processing stage, and combined with the ID corresponding to each data entry in the aggregation index table and the end time of the current aggregation processing stage, the initial start ID and end ID of the next aggregation processing stage are determined. The difference between the initial start ID and the termination ID of the next aggregation processing step is compared with a preset calculation capacity value. If the difference between the initial start ID and the termination ID of the next aggregation processing step is greater than the preset computing capacity value, then the IDs between the initial start ID and the termination ID of the next aggregation processing step are processed in batches so that the number of IDs processed in the next aggregation processing step is less than the preset computing capacity value, and the start ID and termination ID of the next aggregation processing step are updated.

4. The method according to claim 3, characterized in that, The batch processing is performed using the preset calculation capacity value as the number of IDs to be processed in the next aggregation processing stage; Alternatively, the batch processing can be performed by evenly distributing the number of IDs processed in different aggregation processing stages.

5. The method according to claim 1, characterized in that, The step of forming a converged index table for the ID corresponding to each piece of data includes: Based on different preset aggregation processing steps, the ID corresponding to each piece of data is used to form the aggregation index table.

6. The method according to claim 1, characterized in that, Each aggregation processing step is determined by a 1-hour aggregation cycle.

7. A massive data aggregation device, characterized in that, The device includes: The acquisition unit is used to acquire the initial data table currently waiting to be aggregated, wherein different data in the initial data table are stored in the format of daily tables; The sorting unit is used to sort different data in the initial data table according to time based on the timestamp indicated by the initial data table in the daily table format, and to obtain the ID corresponding to each data item, wherein the ID refers to the number obtained by sorting. The determination unit is used to determine the start ID and end ID of the next aggregation processing stage; The aggregation unit is used to locate the corresponding data in the initial data table according to the ID range consisting of the start ID and the end ID of the next aggregation processing step, and to perform the aggregation processing. The sorting unit is specifically used for: Based on the timestamp indicated by the initial data table in the daily table format, the different data in the initial data table are sorted according to time, and the ID corresponding to each data is obtained; The ID corresponding to each piece of data is used to form a convergence index table, wherein the convergence index table identifies the ID corresponding to each piece of data, the maximum ID in the data of the current convergence processing stage, the original table name, the creation time of the data corresponding to the maximum ID in the data of the current convergence processing stage, and the end time of the current convergence processing stage; The determining unit is specifically used for: Determine the start and end times of the next aggregation processing step; Based on the start and end times of the next aggregation processing stage, and combined with the ID corresponding to each data entry in the aggregation index table and the end time of the current aggregation processing stage, the start ID and end ID of the next aggregation processing stage are determined.

8. A processing apparatus, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the method as described in any one of claims 1 to 6 when it invokes the computer program in the memory.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the method of any one of claims 1 to 6.

Images