Data processing method, device, storage medium, and computer program product

Abstract

Embodiments of the present disclosure provide a data processing method, a device, a storage medium, and a computer program product. In the embodiments of the present disclosure, in the process of a first operator performing an aggregation operation on a target data stream, if memory is insufficient, processed data rows in a memory space corresponding to the first operator may overflow to an external storage space, so as to promptly solve the problem of insufficient memory space corresponding to the first operator. After the first operator completes aggregation calculation on the target data stream, an intermediate data stream is constructed on the basis of the overflow processed data rows and processed data rows left in the memory space of the first operator, the processed data rows comprised in the intermediate data stream being each associated with an aggregation value generated by the first operator. The intermediate data stream is inputted to a second operator, so that the second operator can complete aggregation calculation on the basis of the aggregation values comprised in the intermediate data stream, and generate an aggregation result corresponding to the target data stream, which can effectively improve the efficiency of aggregation calculation in the second operator.

Description

A data processing method, apparatus, storage medium, and computer program product Technical Field

[0001] This disclosure relates to the field of computing technology, and in particular to a data processing method, apparatus, storage medium, and computer program product. Background Technology

[0002] A window function is a function that performs real-time analytical processing in a database. It can perform calculations on a set of related rows of data (called a window) and return an additional column for each row of data within the window, which is filled with the calculation results for the rows of data within that window.

[0003] Currently, window functions may encounter memory shortages when processing large-scale data. In such cases, the common approach is to "spill" the data to disk to free up memory. However, after spilling the data to disk, some aggregation results obtained by the window function from performing aggregation calculations on the data are also discarded. This inevitably wastes previous work, requiring recalculation of the original input data, reducing data computation efficiency and resulting in a poor user experience. Summary of the Invention

[0004] This disclosure provides a data processing method, apparatus, storage medium, and computer program product for improving the efficiency of aggregate computing in the context of addressing memory shortage problems.

[0005] This disclosure provides a data processing method, including:

[0006] In response to the aggregation calculation instruction, the target data stream is aggregated using the first operator according to the aggregation function indicated in the aggregation calculation instruction;

[0007] During the aggregation calculation performed by the first operator, if it is detected that the remaining amount of the memory space corresponding to the first operator is lower than a preset standard, then some or all of the processed data rows generated by the first operator in the memory space are spilled to the external storage space, and the aggregated value generated by the first operator is associated with the processed data rows.

[0008] After the first operator completes the aggregation calculation of the target data stream, an intermediate data stream is constructed based on the processed data rows left in the memory space and the processed data rows overflowed for the target data stream.

[0009] The second operator is used to perform aggregation calculations on the aggregated values ​​associated with the data rows contained in the intermediate data stream according to the aggregation function, so as to generate an aggregation result for the target data stream;

[0010] When the second operator is detected to generate any aggregation result during the aggregation calculation process, the memory space occupied by the data row corresponding to the aggregation result in the intermediate data stream under the second operator is released.

[0011] This disclosure also provides a computing device, including: a memory and a processor;

[0012] The memory is used to store one or more computer instructions;

[0013] The processor is coupled to the memory and is used to execute the one or more computer instructions for:

[0014] In response to the aggregation calculation instruction, the target data stream is aggregated using the first operator according to the aggregation function indicated in the aggregation calculation instruction;

[0015] During the aggregation calculation performed by the first operator, if it is detected that the remaining amount of the memory space corresponding to the first operator is lower than a preset standard, then some or all of the processed data rows generated by the first operator in the memory space are spilled to the external storage space, and the aggregated value generated by the first operator is associated with the processed data rows.

[0016] After the first operator completes the aggregation calculation of the target data stream, an intermediate data stream is constructed based on the processed data rows left in the memory space and the processed data rows overflowed for the target data stream.

[0017] The second operator is used to perform aggregation calculations on the aggregated values ​​associated with the data rows contained in the intermediate data stream according to the aggregation function, so as to generate an aggregation result for the target data stream;

[0018] When the second operator is detected to generate any aggregation result during the aggregation calculation process, the memory space occupied by the data row corresponding to the aggregation result in the intermediate data stream under the second operator is released.

[0019] This disclosure also provides a computer-readable storage medium for storing computer instructions, which, when executed by one or more processors, cause the one or more processors to perform the aforementioned data processing method.

[0020] This disclosure also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the processor performs the aforementioned data processing method.

[0021] In this embodiment of the disclosure, if the first operator encounters a memory shortage during the aggregation operation of the target data stream according to the aggregation function indicated in the aggregation calculation instruction, some or all of the processed data rows stored in the memory space corresponding to the first operator can be spilled to the external storage space to release the memory space corresponding to the first operator. This can solve the memory shortage problem of the first operator in a timely manner and ensure that the first operator has sufficient memory during operation.

[0022] After the first operator completes the aggregation calculation on the target data stream, an intermediate data stream is constructed based on the overflowed processed data rows and the remaining processed data rows in the memory space of the first operator. Each processed data row in the intermediate data stream is associated with the aggregation value generated by the first operator. Based on this, the intermediate data stream can be input to the second operator provided in this embodiment. The second operator, based on the aggregation values ​​associated with the data rows in the intermediate data stream, can take over from the first operator and continue to perform aggregation calculations according to the aggregation function to generate the aggregation result corresponding to the target data stream. This effectively improves the efficiency of the aggregation calculation in the second operator.

[0023] Furthermore, the process of the second operator performing aggregation calculations can be monitored. When the second operator generates any aggregation result during the aggregation calculation process, the memory space occupied by the data row corresponding to the aggregation result in the intermediate data stream under the second operator can be released. This provides more available memory space for other data rows in the intermediate data stream, ensuring that the second operator has sufficient memory during operation and avoiding new memory shortage problems introduced by the second operator.

[0024] Accordingly, in this embodiment, not only can the memory shortage problem of the first operator be solved, but the second operator can also make full use of the preceding work done by the first operator while ensuring sufficient memory, thereby completing the continued aggregation process more efficiently and effectively improving the overall aggregation calculation efficiency. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0026] Figure 1 is a flowchart illustrating a data processing method provided in an exemplary embodiment of this disclosure;

[0027] Figure 2 is a logical schematic diagram of a data processing method provided in an exemplary embodiment of this disclosure;

[0028] Figure 3 is a schematic diagram of an intermediate data stream provided by an exemplary embodiment of the present disclosure;

[0029] Figure 4 is a logical schematic diagram of a data processing method provided in another exemplary embodiment of this disclosure;

[0030] Figure 5 is a schematic diagram of the state in the memory space corresponding to the first operator and the second operator respectively, provided in another exemplary embodiment of this disclosure;

[0031] Figure 6 is a schematic diagram of the structure of a computing device provided in another exemplary embodiment of this disclosure. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0033] In existing technologies, when using traditional window function operators to perform aggregation calculations on a target data stream, if insufficient memory is encountered, data is typically spilled to disk to free up memory space. Since the aggregation calculation is not yet complete, the data spilled to disk is the original target data stream. If the target data stream from disk is then injected back into the memory of the traditional window function operator to continue calculation, the memory shortage problem will recur, creating a vicious cycle that prevents the aggregation calculation of the target data stream from being completed. If the operator is changed, some of the aggregation results obtained by the traditional window function operator cannot be used, requiring recalculation of the original input data, which reduces the efficiency of the aggregation calculation and results in a poor user experience.

[0034] To address this issue, some embodiments of this disclosure propose that when the first operator encounters insufficient memory, the processed data rows containing the aggregated values ​​can be spilled to external storage space to promptly release the memory space corresponding to the first operator. After the first operator completes the aggregation calculation on the target data stream, an intermediate data stream is constructed based on the processed data rows spilled for the target data stream and the remaining processed data rows in the memory space. This intermediate data stream is then input to the second operator, allowing the second operator to fully utilize the aggregated values ​​generated by the first operator to complete the aggregation calculation. The preceding work performed by the first operator is no longer wasted, saving time in executing the aggregation calculation and thus effectively improving aggregation efficiency.

[0035] The technical solutions provided by the embodiments of this disclosure are described in detail below with reference to the accompanying drawings.

[0036] Figure 1 is a schematic flowchart of a data processing method provided in an exemplary embodiment of this disclosure. The method can be executed by a data processing device, which can be implemented as a combination of software and / or hardware, and can be integrated into a computing device. Referring to Figure 1, the method includes:

[0037] Step 100: In response to the aggregation calculation instruction, perform aggregation calculation on the target data stream using the first operator according to the aggregation function indicated in the aggregation calculation instruction;

[0038] Step 101: During the aggregation calculation of the first operator, if it is detected that the remaining amount of memory space corresponding to the first operator is lower than the preset standard, then some or all of the processed data rows generated by the first operator in the memory space are spilled to the external storage space, and the aggregated value generated by the first operator is associated with the processed data rows.

[0039] Step 102: After the first operator completes the aggregation calculation of the target data stream, an intermediate data stream is constructed based on the remaining processed data rows in the memory space and the processed data rows overflowed for the target data stream.

[0040] Step 103: Use the second operator to perform aggregation calculations on the aggregated values ​​associated with the data rows contained in the intermediate data stream according to the aggregation function, so as to generate an aggregation result for the target data stream;

[0041] Step 104: When the second operator generates any aggregation result during the aggregation calculation process, release the memory space occupied by the data row corresponding to the aggregation result in the intermediate data stream under the second operator.

[0042] The data processing method provided in this embodiment can be used in various scenarios that require aggregation calculations using window functions. For example, window functions can be used to calculate the cumulative or average values ​​of indicators such as a customer's historical repayment records and loan usage, which can be used as a basis for credit assessment; window functions can also be used to calculate the cumulative sales volume and ranking of each sales team to evaluate the performance of the sales team. Of course, this embodiment is not limited to these.

[0043] Before going into detail, let's briefly explain a few technical concepts involved in this embodiment.

[0044] Window functions are functions used in database queries to perform complex analysis and calculations on result sets. They allow users to sort, rank, and calculate aggregate values ​​on a group of related rows (called a window) without grouping or summarizing the original rows. Unlike traditional aggregate functions that summarize multiple rows into a single result, window functions return a result for each row, thus preserving all the original data.

[0045] Spill: A term in computer science, it generally refers to the process where, when system memory (RAM) is insufficient to hold the data or operations being processed, the results of those data or operations are temporarily transferred to a slower storage medium (such as a hard disk drive or solid-state drive).

[0046] Operators: In computer science, particularly in the context of deep learning and data processing, operators typically refer to functions or modules that perform specific computational tasks. These operators can be independent, reusable logical expressions that can be combined to achieve higher-dimensional processing logic. In data processing and query languages, window functions are also a special type of operator used to perform complex aggregation, sorting, and analysis operations on each row of a result set.

[0047] The data processing method provided in this embodiment innovatively proposes a special operator, namely the second operator, which can be used to continue the aggregation operation on the aggregated value generated by the traditional operator, so that the aggregation work performed by the traditional operator will not be wasted, thereby effectively improving the efficiency of aggregation calculation.

[0048] Based on this, the technical solution for data computation based on this special second operator will be described in detail below.

[0049] Referring to Figure 1, in step 100, in response to the aggregation calculation instruction, a first operator can be used to perform aggregation calculations on the target data stream according to the aggregation function indicated in the aggregation calculation instruction. The first operator is an operator that executes a window function, which can be used in conjunction with an aggregation function to perform aggregation calculations on the target data stream within a window. Aggregation functions are used to calculate a set of data and return the calculation results. The aggregation functions indicated in the aggregation calculation instruction include, but are not limited to: calculating the number of specified elements in a row or column (COUNT function), finding the maximum value in a column (MAX function), finding the minimum value in a column (MIN function), and calculating the sum of values ​​in a column (SUM function), etc.

[0050] There is a difference between using aggregate functions alone and using them in combination with window functions. When used alone, an aggregate function performs aggregation calculations on the target data stream and returns a single aggregation result. When aggregate functions are used in combination with window functions, an aggregation result can be provided for each data row without collapsing the data rows. This means that an aggregation result based on the data within its window can be generated for each data row in the target data stream.

[0051] During the aggregation calculation of the target data stream, the first operator can divide the data rows in the target data stream into blocks (or "windows") according to the characteristics of the window function, dividing the target data stream into multiple data blocks. These multiple data blocks are then sequentially input into the first operator, which performs aggregation calculations on each data block. When windowing the target data stream, the blocks can be divided according to the number of data rows, for example, cutting the target data stream every 1000 data rows to form a new data block; or they can be divided according to time intervals, for example, cutting the target data stream every 5 minutes to form a new data block. Of course, other methods can also be used to divide the target data stream into multiple data blocks; this embodiment does not limit this approach.

[0052] In step 101, during the aggregation calculation performed by the first operator, the remaining memory space corresponding to the first operator can be monitored in real time. This memory space refers to the running memory space allocated to the first operator by the computing engine controlling the entire calculation process. If the remaining memory space corresponding to the first operator is detected to be lower than a preset standard, meaning the currently available memory space for the first operator is insufficient to meet the current computational needs, then some or all of the processed data rows generated by the first operator in the memory space are spilled to external storage. The processed data rows are generated by the first operator during the aggregation calculation of the target data stream.

[0053] As described above, the first operator performs calculations on the target data stream in the form of data blocks. If the remaining space in the memory space corresponding to the first operator is lower than a preset standard, the complete data block stored in the current memory space can be spilled to the external storage space. Spilling some or all of the processed data lines generated by the first operator in the memory space to the external storage space can be understood as follows: if the remaining space in the memory space corresponding to the first operator is detected to be lower than the preset standard, and if the first operator has just finished calculating a data block, then all processed data lines in the memory space are spilled to the external storage space; if the first operator is currently calculating a data block, then some processed data lines in the memory space (i.e., the complete data block stored in the memory space) are spilled to the external storage space. The external storage space can be a disk, solid-state drive, network storage device, etc., and this embodiment does not limit this.

[0054] After the processed data rows are spilled to external storage, the memory space corresponding to the first operator is released, and the first operator can continue to perform aggregation calculations on other data blocks in the target data stream. However, during the aggregation calculation of other data blocks by the first operator, it is still possible to encounter a situation where the remaining memory space is lower than a preset standard. In this case, some or all of the processed data rows generated by the first operator in the memory space will be spilled to external storage again. In the external storage space, the processed data rows spilled each time can be stored in different storage files, or the processed data rows spilled multiple times can be stored in the same storage file. This embodiment does not limit this.

[0055] The processed data rows spilled to external storage space are stored in the form of data blocks. Each processed data row is associated with an aggregate value generated by the first operator. The aggregate value associated with the processed data row is a stage result obtained by performing aggregation calculation on the data block where the processed data row is located, which has not yet reached the final goal. These results are usually only a part of the calculation process and can be used to support subsequent calculations or analysis.

[0056] Figure 2 is a logical schematic diagram of a data processing method provided by an exemplary embodiment of this disclosure. As shown in Figure 2, after the first operator completes the aggregation calculation on the target data stream, an intermediate data stream can be constructed based on the processed data rows remaining in the memory space and the processed data rows overflowed by the target data stream. The constructed intermediate data stream can be used as the input of the second operator for the second operator to perform aggregation calculation on the intermediate data stream.

[0057] In this embodiment, the processed data row obtained after calculation by the first operator is processed into an intermediate data stream. This intermediate data stream can be adapted to the data format of the second operator, breaking the data format incompatibility between the first and second operators. This allows the second operator to make full use of the preceding work done by the first operator, thereby effectively improving the overall efficiency of aggregation calculation while solving the problem of insufficient memory.

[0058] In this embodiment, the processed data rows contained in the intermediate data stream can be arranged in an ordered manner, that is, the data blocks left in the storage space and the data blocks spilled to the external storage space are merged into an ordered linked list.

[0059] In one alternative implementation of constructing the intermediate data stream, a K-way merge algorithm can be used. First, within each data block, the processed data rows are sorted according to the element values ​​in the aggregation column indicated in the aggregation calculation instruction, resulting in multiple ordered data blocks. The first processed data row from each ordered data block is placed into a min-heap, and the data block to which each processed data row belongs is recorded. The processed data row with the smallest element value is retrieved from the min-heap and placed into a linked list. The next processed data row is then retrieved from the data block to which this processed data row belongs and placed into the min-heap. These steps are repeated until all processed data rows are placed into the linked list, ensuring that identical element values ​​are consecutively arranged in the linked list. The intermediate data stream is then constructed based on the final linked list.

[0060] In another alternative implementation of constructing the intermediate data stream, legacy processed data rows and processed data rows spilled to the target data stream can be combined into a data table. This is equivalent to mixing the processed data rows left in memory space with the processed data rows spilled to external storage space to form a single data table. In this data table, the element values ​​in the aggregation columns indicated in the aggregation calculation instructions are compared and sorted so that identical element values ​​are arranged consecutively. The sorting results of the aggregation columns are extended to other columns in the data table to obtain a reordered data table, and the intermediate data stream is constructed based on this reordered data table.

[0061] The aggregation column specified in the aggregation calculation instruction is crucial for performing aggregation calculations. It allows for the execution of corresponding aggregation calculations on the element values ​​of the aggregation column according to the aggregation function specified in the instruction. It also allows for the sorting of processed data rows based on the element values ​​of the aggregation column. For example, when it is necessary to count the number of employees under each employee level, the column corresponding to "Employee Level" in the data stream can be specified as the aggregation column in the aggregation calculation instruction. The element values ​​in the "Employee Level" column can include Level 1, Level 2, Level 3, etc. The processed data rows can be sorted according to the corresponding level values ​​under "Employee Level," or aggregation calculations can be performed on the corresponding level values ​​under "Employee Level" according to the aggregation function.

[0062] Of course, these are just examples, and this embodiment does not limit the way the intermediate data stream is constructed.

[0063] In this embodiment, by constructing an intermediate data stream, processed data rows with the same element value are arranged consecutively. During the aggregation calculation, the second operator does not need to traverse all data rows to obtain all processed data rows associated with a single element value. This allows the second operator to output the aggregation result for a single element value as early as possible and promptly release the corresponding memory to the processed data row associated with another element value, thereby completing the aggregation process more efficiently and effectively improving the efficiency of aggregation calculation.

[0064] In the above or below embodiments, various implementation methods can be used to compare and sort the element values ​​in the aggregate column indicated in the aggregate calculation instruction.

[0065] In one alternative implementation, the processed data rows in the data table can be traversed to compare the element values ​​of the aggregate columns in adjacent processed data rows. If the element value of the earlier processed data row is greater than the element value of the later processed data row, the two processed data rows are swapped in the data table. This process of traversing the processed data rows and comparing the element values ​​of the aggregate columns in adjacent processed data rows continues until no more processed data rows need to be swapped, indicating that the sorting is complete.

[0066] Of course, other implementation methods can be used, and this embodiment does not limit them. For example, a traversal operation can be performed on the data table to select the data row with the largest or smallest element value in the aggregate column from the data rows contained in the data table, and the selected data row can be deleted from the data table after the traversal is completed. The data rows obtained by performing multiple traversal operations on the data table are arranged sequentially to obtain a reordered data table.

[0067] During the research process, the inventors discovered that when performing aggregation calculations on any data block, the same aggregation value is associated on multiple processed data rows. These aggregation values ​​have the same meaning, which may cause the subsequent second operator to repeatedly reference the aggregation value when performing subsequent calculations based on the aggregation value, resulting in inaccurate aggregation results.

[0068] Therefore, this embodiment proposes that when some or all of the processed data rows generated by the first operator in the memory space are spilled to the external storage space, a filter identifier can be added to the specified processed data rows. The filter identifier can be used to guide the second operator to use the aggregate value associated with the identified processed data row for aggregation calculation.

[0069] For example, one possible implementation of adding a filter identifier is as follows: A data block to be overflowed is selected from memory space, containing processed data rows. Within the data block, deduplication is performed on the aggregate values ​​associated with the processed data rows. The processed data rows containing the retained aggregate values ​​after deduplication are designated as the specified processed data rows, and a filter identifier is added to these designated processed data rows. The processed data rows carrying the filter identifier are stored in external storage space as the processed data rows overflowed for the target data stream. In this way, when the second operator performs subsequent aggregation calculations using the aggregate values ​​generated by the first operator, the processed data rows marked with the filter identifier need to be calculated, avoiding repeated references to aggregate values ​​with the same meaning, thereby effectively improving the accuracy of the aggregation results.

[0070] Accordingly, after the first operator completes the aggregation calculation of the target data stream, it can also perform deduplication of the aggregated values ​​in each data block remaining in the memory space, and add a filter flag to the processed data row containing the retained aggregated values ​​after deduplication. Here, the deduplication operation does not delete duplicate data rows, but rather deletes duplicate aggregated values. Similarly, the second operator needs to calculate the processed data rows marked with filter flags, avoiding repeated references to aggregated values ​​with the same meaning, thereby effectively improving the accuracy of the aggregation results.

[0071] Based on this, the remaining processed data rows carrying filter identifiers and the processed data rows carrying filter identifiers spilled to the target data stream can be combined into a data table. It's important to note that this doesn't involve combining only the processed data rows marked with filter identifiers; rather, it involves combining all processed data rows remaining in the internal space and all processed data rows spilled to external storage, except for some processed data rows marked with filter identifiers. In this data table, during the comparison and sorting of elements in the aggregation columns indicated by the aggregation calculation instructions, the filter identifiers also move along with the processed data rows, remaining closely associated with the processed data rows they mark.

[0072] Figure 3 is a schematic diagram of an intermediate data stream provided by an exemplary embodiment of this disclosure. As shown in Figure 3, in this embodiment, the filter flag can be further divided into two types: true and false. A filter flag of true is added to the processed data rows that retain the aggregate value after deduplication, and a filter flag of false is added to the processed data rows that delete the aggregate value after deduplication. When the second operator performs subsequent aggregation calculations using the aggregate value generated by the first operator, it needs to perform calculations on the processed data rows marked with true.

[0073] Based on this, the intermediate data stream can be input into the second operator. The intermediate data stream is input into the second operator in a streaming manner.

[0074] Referring again to Figure 2, after the intermediate data stream is input into the second operator, the second operator can perform aggregation calculations on the aggregated values ​​associated with the data rows contained in the intermediate data stream according to the aggregation function, thereby generating an aggregation result for the target data stream. The aggregation function followed by the second operator is the same as that followed by the first operator, and the intermediate data stream constructed based on the processed data rows generated by the first operator is compatible with the data format of the second operator. Therefore, the second operator can continue to perform aggregation calculations based on the aggregated values ​​generated by the first operator. The aggregation result calculated by the second operator is the result obtained by continuing to perform aggregation calculations on the aggregated values ​​generated by the first operator.

[0075] Referring again to Figure 1, in step 104, the process of the second operator performing aggregation calculations can be monitored. When it is detected that the second operator generates any aggregation result during the aggregation calculation process, the memory space occupied by the data row corresponding to the aggregation result in the intermediate data stream under the second operator is released. The released memory space can provide more available memory space for other data rows in the intermediate data stream, ensuring sufficient memory for the second operator during operation and thus guaranteeing the smooth operation of the second operator.

[0076] Optionally, during the aggregation calculation of the aggregated values ​​associated with the data rows contained in the intermediate data stream using the second operator, if it is determined that the target element value in the aggregation column meets the release condition, then it is determined that an aggregation result has been generated for the processed data row containing the target element value. The memory occupied by the data row containing the target element value can then be released from the memory space corresponding to the second operator, and the aggregation result corresponding to the target element value can be output. Here, the release condition means that the processed data row containing the target element value has been calculated, an aggregation result has been generated for the processed data row containing the target element value, and subsequent aggregation calculations no longer require the use of the processed data row containing the target element value.

[0077] Accordingly, in this embodiment, an intermediate data stream is constructed based on the processed data rows generated by the first operator, and a second operator is used to perform aggregation calculations on the intermediate data stream. This allows the second operator to fully utilize the aggregated values ​​associated with the processed data rows generated by the first operator to complete the aggregation calculations. This ensures that the preliminary work performed by the first operator is not wasted, saving time on aggregation calculations and effectively improving aggregation efficiency. Furthermore, during the aggregation calculation process of the second operator, the timely release of the second operator's memory space is further optimized, effectively avoiding the problem of insufficient memory for the second operator.

[0078] In the above or below embodiments, various implementation methods can be used to perform aggregation calculations on the aggregated values ​​associated with the data rows contained in the intermediate data stream using the second operator.

[0079] Figure 4 is a logical schematic diagram of a data processing method provided in another exemplary embodiment of this disclosure. Referring to Figure 4, after the intermediate data stream is input to the second operator, the second operator can divide the received intermediate data stream into data blocks according to a preset data volume, and store the divided data blocks in the memory space of the second operator. That is, the data block serves as the storage unit in the memory space of the second operator. Since the intermediate data stream is input to the second operator in a streaming manner, the second operator will also adapt to this streaming input method when dividing the data blocks.

[0080] The process of the second operator dividing the intermediate data stream into data blocks can be similar to the process of the first operator dividing the target data stream into data blocks, but the sizes of the data blocks divided by the first operator and the data blocks divided by the second operator can be different. For example, the first operator and the second operator can divide the data stream into blocks according to different numbers of data rows or different time intervals.

[0081] After dividing the received intermediate data stream into data blocks, the element values ​​under the aggregate column in different processed data rows within any data block may be different. A second operator can be used to further divide the data blocks, and the resulting data blocks are then successively divided into data partitions. Processed data rows with the same element value under the aggregate column within the same data block are assigned to the same data partition.

[0082] As mentioned above, data blocks can be used as storage units in the memory space of the second operator, and similarly, data blocks can also be used as release units for memory release in the second operator. Furthermore, when performing aggregation calculations on a data partition basis, it is possible to determine whether the memory occupied by the data block to which the data partition belongs can be released, rather than releasing memory on a data partition basis.

[0083] Optionally, in one implementation of dividing a data block into data partitions, it can be determined whether the element values ​​under the aggregate column in the current processed data row and the element values ​​under the aggregate column in the next processed data row have the same semantic meaning within the current data block. If they do, the next processed data row is taken as the current processed data row, and the determination of whether the element values ​​under the aggregate column in adjacent processed data rows have the same semantic meaning continues until a case of semantic difference occurs or it is determined that the element values ​​corresponding to the aggregate column in each processed data row within the current data block have the same semantic meaning. If they do not have the same semantic meaning, the current processed data row is used as a dividing line to divide the current processed data row and the processed data rows before it into the same data partition; the determination of the processed data rows after the current processed data row in the current data block continues until the determination of the last processed data row in the current data block is completed.

[0084] In this implementation, when determining whether the element values ​​corresponding to the aggregated columns in two adjacent processed data rows are semantically identical, the criteria for determining "semantic similarity" can be flexibly configured according to actual needs. For example, when the element value is a number, it can be configured that if two element values ​​are exactly the same, then the two element values ​​are considered semantically identical; when the element value is text, it can be configured that if the semantic similarity between two element values ​​is greater than a preset threshold, then the two element values ​​are considered semantically identical. Of course, these are exemplary, and this embodiment does not limit them.

[0085] For example, if the intermediate data stream received by the second operator contains 10,000 processed data rows, where the element value under the aggregation column in rows 1-2000 is 1, the element value under the aggregation column in rows 2001-7000 is 2, the element value under the aggregation column in rows 7001-9000 is 3, and the element value under the aggregation column in rows 9001-10000 is 4, then after inputting the intermediate data stream into the second operator, the second operator can be used to divide every 3,000 processed data rows into a data block, thus dividing the intermediate data stream into 4 data blocks. The processed data rows can be numbered to represent the divided data blocks as: [1-3000], [3001-6000], [6001-9000], [9001-10000]. Although the last 1000 rows do not meet the requirement of 3000 rows, they are automatically grouped into a data block since there are no more data rows after them. Based on this, the second operator is used to further divide these 4 data blocks, resulting in the following 4 data blocks: [1*(1-2000),2*(2001-3000)], [2*(3001-6000)], [2*(6001-7000),3*(7001-9000)], [4*(9001-10000)]. Taking the first data block [1*(1-2000),2*(2001-3000)] as an example, the meaning is explained as follows: This data block contains two data partitions. The first data partition "1*(1-2000)" is composed of the 1st to 2000th processed data rows with an element value of 1, and the second data partition "2*(2001-3000)" is composed of the 2nd to 3000th processed data rows with an element value of 2.

[0086] Based on this, the intermediate data stream is divided into data blocks, and then the data blocks are further divided into data partitions. The data partitions can be used as the aggregation calculation units of the second operator, and the aggregation calculations can be performed sequentially within the divided data partitions using the second operator.

[0087] Figure 5 is a schematic diagram of the state within the memory space corresponding to the first operator and the second operator, provided in another exemplary embodiment of this disclosure. As shown in Figure 5, during the process of performing aggregation calculations within a data partition using the second operator, the memory occupied by data blocks that meet the release conditions can be released from the memory space of the second operator in a timely manner. In this embodiment, various implementation methods can be used to determine whether the target element value in the aggregation column meets the release conditions.

[0088] In an alternative implementation, a second operator can be used to determine whether the current data block contains multiple data partitions.

[0089] If the current data block contains multiple data partitions, after the second operator completes the aggregation calculation for the first data partition in the current data block, it can be determined that the target element value corresponding to the first data partition in the aggregation column meets the release condition, and the memory occupied by the data blocks before the current data block can be released from the memory space corresponding to the second operator.

[0090] If the current data block contains a data partition, after the second operator completes the aggregation calculation for the first data partition within the current data block, it determines whether the target element value corresponding to the first data partition in the aggregation column is the same as the element value corresponding to the first data partition in the aggregation column within the data blocks following the current data block. If they are different, it is determined that the target element value corresponding to the first data partition in the aggregation column meets the release condition, and the memory occupied by the current data block and the data blocks preceding the current data block is released from the memory space corresponding to the second operator; if they are the same, the aggregation calculation continues to be performed on the data partitions within the data blocks following the current data block.

[0091] In this implementation, the release condition is associated with the first data partition in a data block containing multiple data partitions. After the second operator completes the aggregation calculation on the first data partition, the memory occupied by the data block before the current data block is released. This eliminates the need to repeatedly check whether the element value meets the release condition, reducing the workload of the second operator. The Shudie second operator can focus on performing aggregation calculations within the data partition, thereby further improving the efficiency of aggregation calculations.

[0092] To further improve the timeliness of memory release, after the second operator completes the aggregation calculation for the last data partition within the current data block, it can be determined whether the target element value corresponding to the last data partition in the aggregation column is the same as the element value corresponding to the first data partition in the aggregation column within the data block following the current data block. If they are different, it is determined that the target element value corresponding to the last data partition in the aggregation column meets the release condition, and the memory occupied by the current data block is released from the memory space corresponding to the second operator; if they are the same, the aggregation calculation continues to be performed on the data partitions within the data blocks following the current data block.

[0093] Continuing with the four data blocks from the example above: [1*(1-2000), 2*(2001-3000)], [2*(3001-6000)], [2*(6001-7000), 3*(7001-9000)], and [4*(9001-10000)], the implementation logic for memory release will be described below:

[0094] 1. After the second operator completes the aggregation calculation on the first data partition of the first data block, since the first data block is the first data block, there is no need to release memory. The second operator then continues to perform aggregation calculation on the second data partition of the first data block. After completing the aggregation calculation on the second data partition, it is determined whether the element values ​​in the second data partition are the same as the element values ​​contained in the first data partition of the second data block. The determination confirms that the element values ​​in the two data partitions are the same; therefore, the memory occupied by the first data block in the memory space of the second operator cannot be released.

[0095] 2. Continue the aggregation calculation on the data partition in the second data block using the second operator. The second data block contains one data partition. After completing the aggregation calculation on this data partition, determine whether the element value in this data partition is the same as the element value contained in the first data partition in the third data block. It is determined that the element values ​​in the two data partitions are the same. Therefore, the memory occupied by the second data block in the memory space of the second operator cannot be released.

[0096] 3. Continue the aggregation calculation on the data partitions in the third data block using the second operator. The third data block contains multiple data partitions. After completing the aggregation calculation on the first data partition in the third data block, release the memory occupied by the first and second data blocks in the memory space of the second operator. Continue the aggregation calculation on the second data partition in the same data block using the second operator. After completing the aggregation calculation on the second data partition, determine whether the element values ​​in this data partition are the same as the element values ​​contained in the first data partition in the fourth data block. It is determined that the element values ​​in the two data partitions are different; therefore, the memory occupied by the third data block in the memory space of the second operator can be released.

[0097] 4. Continue to perform aggregation calculations on the data partitions in the fourth data block for the second data partition. Since the fourth data block is the last data block contained in the current data stream, the memory occupied by the remaining data blocks in the memory space of the second operator can be released after the aggregation calculations on the data partitions in the fourth data block are completed.

[0098] It's worth noting that data partitioning is based on element values. Therefore, the release condition can be checked at the partitioning points. If the release condition is met at any partitioning point, the memory occupied by the data block before that partitioning point is released from the memory space corresponding to the second operator. After releasing the memory occupied by the data block, the aggregation result for that data block can be output. However, data partitioning can be used to determine whether the memory occupied by a data block can be released, but memory cannot be released on a per-partition basis.

[0099] In another alternative implementation, the data partitions contained in multiple data blocks can be classified, and at least one data partition with the same element value under the aggregation column can be grouped into the same partition set. Based on this, the pointer contained in the second operator can be used to determine whether the target element value of the current data partition in the aggregation column meets the release condition. The second operator may include a first pointer and a second pointer. The first pointer is used to point to the processed data row participating in the aggregation calculation, and the second pointer is used to point to the first processed data row in the next partition set of the partition set to which the current data partition belongs. During the aggregation calculation of the first data partition in the current data block using the second operator, the execution position of the first pointer and the pointing position of the second pointer can be monitored. If the pointing positions of the first pointer and the second pointer are found to coincide, it is determined that the target element value of the current data partition in the aggregation column meets the release condition.

[0100] In this optional implementation, the release condition is determined in real time by monitoring the positional relationship between the two pointers. This simplifies the logic for determining the release condition and allows for quick and accurate determination of whether the target element value meets the release condition. In this way, unnecessary memory can be released in a timely manner when the release condition is met, thereby effectively avoiding the situation of insufficient memory in the second operator.

[0101] Of course, other implementation methods can be used to determine whether the target element in the aggregate column meets the release conditions, and this embodiment does not limit this.

[0102] Based on this, if the second operator has finished calculating the current data block and the intermediate data stream has not been fully input, the intermediate data stream can continue to be streamed into the second operator. This allows the second operator to divide the newly input intermediate data stream into data blocks and partitions, and then continue performing aggregation calculations on the newly partitioned data. If the second operator has finished calculating the current data block and no new data blocks are input, it means that the aggregation calculation of the intermediate data stream is nearing completion. After completing the aggregation calculation, the remaining aggregation results can be directly output, and the memory occupied by all data blocks can be released. In this way, unnecessary memory can be released as early as possible during the aggregation calculation process, and the aggregation results can be output as early as possible. Furthermore, timely release of unnecessary memory can effectively avoid memory shortages, thus achieving a memory-friendly effect.

[0103] In addition, other implementation methods can be used to perform aggregation calculations on the aggregated values ​​associated with the data rows contained in the intermediate data stream using the second operator. For example, the second operator can be used to divide the intermediate data stream into different data blocks according to the element values ​​in the aggregation column indicated in the aggregation calculation instruction, with each data block containing one type of element value. After completing the aggregation calculation for the current data block, the memory occupied by the current data block is released from the memory space corresponding to the second operator. This embodiment does not limit this approach.

[0104] The execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 100 to 102 can be device A; or the execution subject of steps 100 and 101 can be device A, and the execution subject of step 104 can be device B; and so on.

[0105] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. The terms "first," "second," etc., used herein are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0106] Figure 6 is a schematic diagram of a computing device provided in another exemplary embodiment of this disclosure. As shown in Figure 6, the computing device includes a memory 60 and a processor 61.

[0107] Processor 61, coupled to memory 60, is used to execute computer programs in memory 60 for:

[0108] In response to the aggregation calculation instruction, the first operator performs aggregation calculation on the target data stream according to the aggregation function indicated in the aggregation calculation instruction;

[0109] During the aggregation calculation of the first operator, if it is detected that the remaining amount of memory space corresponding to the first operator is lower than the preset standard, then some or all of the processed data rows generated by the first operator in the memory space will be spilled to the external storage space, and the aggregated value generated by the first operator will be associated with the processed data rows.

[0110] After the first operator completes the aggregation calculation of the target data stream, an intermediate data stream is constructed based on the remaining processed data rows in the memory space and the processed data rows that have been overflowed for the target data stream.

[0111] The second operator is used to perform aggregation calculations on the aggregated values ​​associated with the data rows contained in the intermediate data stream according to the aggregation function, so as to generate an aggregation result for the target data stream;

[0112] When the second operator generates any aggregation result during the aggregation calculation process, the memory space occupied by the data row corresponding to the aggregation result in the intermediate data stream under the second operator is released.

[0113] In an optional embodiment, during the process of constructing an intermediate data stream based on the remaining processed data lines in memory space and the processed data lines overflowed for the target data stream, the processor 61 is further configured to:

[0114] Combine the remaining processed data rows and the processed data rows overflowed for the target data stream into a data table;

[0115] In a data table, the element values ​​in the aggregate column indicated in the aggregate calculation instruction are compared and sorted so that the same element values ​​are arranged consecutively;

[0116] Extend the sorting results of the aggregated columns to other columns in the data table to obtain a reordered data table;

[0117] An intermediate data stream is constructed based on the reordered data table.

[0118] In an optional embodiment, during the process of spilling some or all of the processed data lines generated by the first operator in the memory space to the external storage space, the processor 61 is further configured to:

[0119] Select a data block to be overflowed from the memory space. The data block to be overflowed contains the processed data rows.

[0120] Within the data block to be overflowed, perform deduplication on the aggregate values ​​associated with the processed data rows;

[0121] Add a filter flag to the processed data row containing the aggregated values ​​that are retained after deduplication;

[0122] The processed data rows carrying the filter identifier are stored in the external storage space as processed data rows spilled for the target data stream;

[0123] The filter identifier is used to guide the second operator to perform aggregation calculations using the aggregated values ​​associated with the identified processed data rows.

[0124] In an optional embodiment, during the process of combining the remaining processed data rows and the processed data rows overflowed for the target data stream into a data table, the processor 61 is further configured to:

[0125] Within each data block remaining in the memory space, perform deduplication operations on the aggregated values.

[0126] Add a filter flag to the processed data row containing the aggregated values ​​that are retained after deduplication;

[0127] Combine the remaining processed data rows carrying the filter flags and the processed data rows overflowed for the target data stream into a data table.

[0128] In an optional embodiment, during the process of comparing and sorting the element values ​​in the aggregation column indicated in the aggregation calculation instruction, the processor 61 is further configured to:

[0129] Iterate through the processed data rows in the data table to compare the element values ​​of the aggregate column in each of the two adjacent processed data rows;

[0130] If the element value of the first processed data row is greater than the element value of the second processed data row, then swap the positions of the two processed data rows in the data table.

[0131] In an optional embodiment, during the process of generating an aggregation result for the target data stream by using a second operator to aggregate the aggregated values ​​associated with the data rows contained in the intermediate data stream according to the aggregation function, the processor 61 is further configured to:

[0132] During the process of using the second operator to perform aggregation calculations on the aggregated values ​​associated with the data rows contained in the intermediate data stream, if it is determined that the target element value in the aggregation column meets the release condition, the memory occupied by the processed data row where the target element value is located is released from the memory space corresponding to the second operator.

[0133] Output the aggregation result corresponding to the target element value.

[0134] In an optional embodiment, during the process of performing aggregation operations on the aggregated values ​​associated with the data rows contained in the intermediate data stream using the second operator according to the aggregation function, the processor 61 is further configured to:

[0135] The second operator is used to divide the received intermediate data stream into data blocks according to a preset data volume.

[0136] Using the second operator, the obtained data block is divided into data partitions in turn. The data partition serves as the aggregation calculation unit of the second operator. The processed data rows with the same element value under the aggregation column within the data block are divided into the same data partition.

[0137] Using the second operator, aggregate calculations are performed sequentially within the divided data partitions.

[0138] In an optional embodiment, if it is determined that the target element value in the aggregate column meets the release condition, during the process of releasing the memory occupied by the processed data row where the target element value is located from the memory space corresponding to the second operator, the processor 61 is further configured to:

[0139] Use the second operator to determine whether the current data block contains multiple data partitions;

[0140] If so, after the second operator completes the aggregation calculation of the first data partition in the current data block, it is determined that the target element value corresponding to the first data partition in the aggregation column meets the release condition;

[0141] Release the memory occupied by the data blocks preceding the current data block from the memory space corresponding to the second operator.

[0142] In an optional embodiment, the second operator further includes a first pointer and a second pointer, and the processor 61 is further configured to:

[0143] During the process of performing aggregation calculations on the first data partition within the current data block using the second operator, the position pointed to by the first pointer is monitored. The first pointer is used to point to the processed data row that is participating in the aggregation calculation.

[0144] Monitor the position of the second pointer, which is used to point to the first processed data row in the next partition set of the partition set to which the current data partition belongs;

[0145] If the first and second pointers are detected to be pointing to the same position, it is determined that the target element value of the current data partition in the aggregate column meets the release condition;

[0146] Each partition set contains at least one data partition that has the same element value under the aggregation column.

[0147] Furthermore, as shown in Figure 6, the computing device also includes other components such as a communication component 62, a display 63, and a power supply component 64. Figure 6 schematically shows only some of the components and does not imply that the computing device includes only the components shown in Figure 6.

[0148] It is worth noting that the technical details of the above embodiments of the computing device can be referred to the relevant descriptions of the operation of the computing device in the aforementioned embodiments of the data processing method. To save space, they will not be repeated here, but this should not cause any loss of the scope of protection of this disclosure.

[0149] This disclosure also provides a computer program product, including a computer program. When the computer program is executed by a processor, it can implement the steps in the above-described data processing method. The technical solution of this computer program and the technical solution of the above-described data processing method belong to the same concept. Details not described in detail in the technical solution of the computer program can be found in the description of the technical solution of the above-described data processing method.

[0150] Accordingly, this disclosure also provides a computer-readable storage medium storing a computer program, which, when executed, can perform the steps that can be executed by a computing device in the above method embodiments.

[0151] The memory shown in Figure 6 above is used to store computer programs and can be configured to store various other data to support operation on the computing platform. Examples of this data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0152] The communication component in Figure 6 above is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA) technology, Ultra-Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0153] The display in Figure 6 above includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action, but also the duration and pressure associated with the touch or swipe operation.

[0154] The power supply component in Figure 6 above provides power to various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0155] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0157] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0158] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0159] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0160] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0161] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0162] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0163] The above descriptions are embodiments of this disclosure and are not intended to limit the scope of this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of the claims of this disclosure.

Claims

1. A data processing method, comprising: In response to the aggregation calculation instruction, the target data stream is aggregated using the first operator according to the aggregation function indicated in the aggregation calculation instruction; During the aggregation calculation performed by the first operator, if it is detected that the remaining amount of the memory space corresponding to the first operator is lower than a preset standard, then some or all of the processed data rows generated by the first operator in the memory space are spilled to the external storage space, and the aggregated value generated by the first operator is associated with the processed data rows. After the first operator completes the aggregation calculation of the target data stream, an intermediate data stream is constructed based on the processed data rows left in the memory space and the processed data rows overflowed for the target data stream. The second operator is used to perform aggregation calculations on the aggregated values ​​associated with the data rows contained in the intermediate data stream according to the aggregation function, so as to generate an aggregation result for the target data stream; When the second operator is detected to generate any aggregation result during the aggregation calculation process, the memory space occupied by the data row corresponding to the aggregation result in the intermediate data stream under the second operator is released.

2. The method according to claim 1, wherein an intermediate data stream is constructed based on the remaining processed data rows in the memory space and the processed data rows overflowed for the target data stream, comprising: The remaining processed data rows and the processed data rows overflowed for the target data stream are combined into a data table; In the data table, the element values ​​in the aggregation column indicated in the aggregation calculation instruction are compared and sorted so that the same element values ​​are arranged consecutively. The sorting results of the aggregated column are extended to other columns in the data table to obtain a reordered data table; Based on the reordered data table, the intermediate data stream is constructed.

3. The method according to claim 2, wherein partially or entirely of the processed data lines generated by the first operator in the memory space are spilled to external storage space, comprising: Select a data block to be overflowed from the memory space, the data block containing the processed data rows; Within the data block to be overflowed, the aggregate values ​​associated with the processed data rows are deduplicated. Add a filter flag to the processed data row containing the aggregated values ​​that are retained after deduplication; The processed data rows carrying the filter identifier are stored in the external storage space as processed data rows overflowed for the target data stream; The filter identifier is used to guide the second operator to perform aggregation calculations using the aggregated values ​​associated with the identified processed data rows.

4. The method according to claim 3, wherein the remaining processed data rows and the processed data rows overflowed for the target data stream are combined into a data table, comprising: Within each data block remaining in the memory space, a deduplication operation is performed on the aggregated value. Add a filter flag to the processed data row containing the aggregated values ​​that are retained after deduplication; The remaining processed data rows carrying the filter identifier and the processed data rows overflowed for the target data stream are combined into a data table.

5. The method according to claim 2, comprising comparing and sorting the element values ​​in the aggregation column indicated in the aggregation calculation instruction, including: Traverse the processed data rows in the data table to compare the element values ​​of the aggregate column in each of two adjacent processed data rows; If the element value of the preceding processed data row is greater than the element value of the following processed data row, then the positions of the two processed data rows in the data table are swapped.

6. The method according to claim 3, wherein the second operator is used to perform aggregation calculations on the aggregation values ​​associated with the data rows contained in the intermediate data stream according to the aggregation function, so as to generate an aggregation result for the target data stream, comprising: During the process of using the second operator to perform aggregation calculations on the aggregated values ​​associated with the data rows contained in the intermediate data stream, if it is determined that the target element value in the aggregation column meets the release condition, then the memory occupied by the processed data row where the target element value is located is released from the memory space corresponding to the second operator. Output the aggregation result corresponding to the target element value.

7. The method according to claim 6, wherein the aggregation operation is performed on the aggregated values ​​associated with the data rows contained in the intermediate data stream using the second operator according to the aggregation function, comprising: The second operator is used to divide the received intermediate data stream into data blocks according to a preset data volume level; Using the second operator, the obtained data block is further divided into data partitions. The data partition serves as the aggregation calculation unit of the second operator. The processed data rows with the same element value under the aggregation column in the data block are divided into the same data partition. Using the second operator, aggregate calculations are performed sequentially within the divided data partitions.

8. The method according to claim 7, wherein if it is determined that the target element value in the aggregate column meets the release condition, the memory occupied by the processed data row where the target element value is located is released from the memory space corresponding to the second operator, comprising: The second operator is used to determine whether the current data block contains multiple data partitions; If so, after the second operator completes the aggregation calculation of the first data partition in the current data block, it is determined that the target element value corresponding to the first data partition in the aggregation column meets the release condition; Release the memory occupied by the data blocks preceding the current data block from the memory space corresponding to the second operator.

9. The method according to claim 8, wherein the second operator further comprises a first pointer and a second pointer, and the method further comprises: During the process of performing aggregation calculation on the first data partition within the current data block using the second operator, the position pointed to by the first pointer is monitored. The first pointer is used to point to the processed data row that is participating in the aggregation calculation. Monitor the position pointed to by the second pointer, which is used to point to the first processed data row in the next partition set of the partition set to which the current data partition belongs; If the first pointer and the second pointer are detected to be pointing to the same position, it is determined that the target element value of the current data partition in the aggregate column meets the release condition; Each partition set contains at least one data partition that has the same element value under the aggregate column.

10. A computing device, comprising a memory and a processor; The memory is used to store one or more computer instructions; The processor is coupled to the memory and is used to execute the one or more computer instructions for: In response to the aggregation calculation instruction, the target data stream is aggregated using the first operator according to the aggregation function indicated in the aggregation calculation instruction; During the aggregation calculation performed by the first operator, if it is detected that the remaining amount of the memory space corresponding to the first operator is lower than a preset standard, then some or all of the processed data rows generated by the first operator in the memory space are spilled to the external storage space, and the aggregated value generated by the first operator is associated with the processed data rows. After the first operator completes the aggregation calculation of the target data stream, an intermediate data stream is constructed based on the processed data rows left in the memory space and the processed data rows overflowed for the target data stream. The second operator is used to perform aggregation calculations on the aggregated values ​​associated with the data rows contained in the intermediate data stream according to the aggregation function, so as to generate an aggregation result for the target data stream; When the second operator is detected to generate any aggregation result during the aggregation calculation process, the memory space occupied by the data row corresponding to the aggregation result in the intermediate data stream under the second operator is released.

11. The computing device of claim 10, wherein, The processor is also used to perform the following methods: The remaining processed data rows and the processed data rows overflowed for the target data stream are combined into a data table; In the data table, the element values ​​in the aggregation column indicated in the aggregation calculation instruction are compared and sorted so that the same element values ​​are arranged consecutively. The sorting results of the aggregated column are extended to other columns in the data table to obtain a reordered data table; Based on the reordered data table, the intermediate data stream is constructed.

12. The computing device of claim 11, wherein, The processor is also used to perform the following methods: Select a data block to be overflowed from the memory space, the data block containing the processed data rows; Within the data block to be overflowed, the aggregate values ​​associated with the processed data rows are deduplicated. Add a filter flag to the processed data row containing the aggregated values ​​that are retained after deduplication; The processed data rows carrying the filter identifier are stored in the external storage space as processed data rows overflowed for the target data stream; The filter identifier is used to guide the second operator to perform aggregation calculations using the aggregated values ​​associated with the identified processed data rows.

13. The computing device of claim 12, wherein, The processor is also used to perform the following methods: Within each data block remaining in the memory space, a deduplication operation is performed on the aggregated value. Add a filter flag to the processed data row containing the aggregated values ​​that are retained after deduplication; The remaining processed data rows carrying the filter identifier and the processed data rows overflowed for the target data stream are combined into a data table.

14. The computing device of claim 12, wherein, The processor is also used to perform the following methods: Traverse the processed data rows in the data table to compare the element values ​​of the aggregate column in each of two adjacent processed data rows; If the element value of the preceding processed data row is greater than the element value of the following processed data row, then the positions of the two processed data rows in the data table are swapped.

15. The computing device of claim 13, wherein, The processor is also used to perform the following methods: During the process of using the second operator to perform aggregation calculations on the aggregated values ​​associated with the data rows contained in the intermediate data stream, if it is determined that the target element value in the aggregation column meets the release condition, then the memory occupied by the processed data row where the target element value is located is released from the memory space corresponding to the second operator. Output the aggregation result corresponding to the target element value.

16. The computing device of claim 15, wherein, The processor is also used to perform the following methods: The second operator is used to divide the received intermediate data stream into data blocks according to a preset data volume level; Using the second operator, the obtained data block is further divided into data partitions. The data partition serves as the aggregation calculation unit of the second operator. The processed data rows with the same element value under the aggregation column in the data block are divided into the same data partition. Using the second operator, aggregate calculations are performed sequentially within the divided data partitions.

17. The computing device of claim 16, wherein, The processor is also used to perform the following methods: The second operator is used to determine whether the current data block contains multiple data partitions; If so, after the second operator completes the aggregation calculation of the first data partition in the current data block, it is determined that the target element value corresponding to the first data partition in the aggregation column meets the release condition; Release the memory occupied by the data blocks preceding the current data block from the memory space corresponding to the second operator.

18. The computing device according to claim 17, wherein, The processor is also used to perform the following methods: During the process of performing aggregation calculation on the first data partition within the current data block using the second operator, the position pointed to by the first pointer is monitored. The first pointer is used to point to the processed data row that is participating in the aggregation calculation. Monitor the position of the second pointer, which is used to point to the first processed data row in the next partition set of the partition set to which the current data partition belongs; If the first pointer and the second pointer are detected to be pointing to the same position, it is determined that the target element value of the current data partition in the aggregate column meets the release condition; Each partition set contains at least one data partition that has the same element value under the aggregate column.

19. A computer-readable storage medium storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the data processing method according to any one of claims 1-9.

20. A computer program product comprising a computer program, wherein, When a computer program is executed by a processor, the processor is caused to perform the data processing method according to any one of claims 1-9.

Images