Grouped data processing method and device based on distributed database KaiwuDB, and medium

Abstract

The application discloses a grouping data processing method and device based on a distributed database KaiwuDB and a medium, belongs to the technical field of databases, and aims to solve the technical problem of how to increase the data processing capacity of a database and improve data processing efficiency. The technical scheme is as follows: an execution process of an orderedAggregator is used, a grouping window function is nested in the orderedAggregator, and the values of data row grouping columns are replaced with a window number before the orderedAggregator performs actual grouping; after the values of the grouping columns in the row data are replaced with the window number, the orderedAggregator is used to group according to the grouping characteristics; and the sorted data is subjected to heap sorting in a gateway node according to a time column, one row of data can be sent to an upper orderedAggregator after each row is sorted, and the orderedAggregator performs grouping window function calculation on the input ordered data.

Description

Technical Field

[0001] This invention relates to the field of database technology, specifically to a method, device, and medium for processing grouped data based on the distributed database KaiwuDB. Background Technology

[0002] A distributed database refers to a cluster composed of multiple database nodes, where data can be stored across multiple nodes. KaiwuDB is a distributed multi-model database designed for AIoT (Artificial Intelligence of Things) scenarios. KaiwuDB's data processing methods include distributed architecture and horizontal scaling, a multi-model fusion engine, time-series data engine optimization, native AI integration, efficient compression and tiered cold / hot storage, transaction processing and consistency guarantees, and out-of-order data processing and deduplication. However, KaiwuDB's current data processing methods have the following shortcomings:

[0003] ① Relatively high transaction latency: Although transaction latency has been reduced through optimization strategies, decentralized design may still result in a high transaction rollback rate and rollback cost in high-concurrency and high-conflict scenarios.

[0004] ② High learning cost: Features such as multi-modal fusion and native AI integration increase the complexity of the system and require higher technical skills from development and maintenance personnel.

[0005] ③ High resource consumption: The design of storing locks and data together may cause each write operation to generate two to three times the write pressure, which may affect performance in resource-constrained environments.

[0006] ④ Adaptability to specific scenarios: Although time-series data processing has been optimized, some advanced features may not be able to fully leverage their advantages in non-AIoT scenarios or low-concurrency scenarios.

[0007] Therefore, how to increase the database's data processing capabilities and improve data processing efficiency is a technical problem that urgently needs to be solved. Summary of the Invention

[0008] The technical objective of this invention is to provide a grouped data processing method, device, and medium based on the distributed database KaiwuDB, in order to solve the problem of how to increase the database's data processing capabilities and improve data processing efficiency.

[0009] The technical task of this invention is achieved as follows: a grouped data processing method based on the distributed database KaiwuDB. This method utilizes the execution flow of orderedAggregator, nesting the grouping window function within orderedAggregator. Before orderedAggregator performs actual grouping, the values ​​of the grouping columns in the data rows are replaced with a window number. After the values ​​of the grouping columns in the row data are replaced with the window number, the grouping characteristics of orderedAggregator are used for grouping. Then, the data is sorted according to the time column in each node of the table data distribution. The sorted data is then heap-sorted at the gateway node. Each row can be sent to the upper-level orderedAggregator. orderedAggregator performs grouping window function calculations on the input ordered data.

[0010] As a preferred method, grouping is performed using the orderedAggregator grouping feature, as follows:

[0011] The grouping window column value for each row is set to groupValue, which starts from 0. Each time a new window is entered, groupValue is incremented by 1 to ensure that the grouping column values ​​for different groups processed by orderedAggregator are different, thereby achieving grouping.

[0012] As a preferred method, the grouping window function `count_window(count_val[,sliding_val])` divides the window according to the number of data rows `count_val`. By default, the data is sorted by the timestamp column. When the cumulative number of data rows reaches `count_val`, the data from row 1 to row `count_val` is grouped together for aggregation calculation. This process is repeated until all data is read and grouped according to `count_val`. Here, `count_val` represents the maximum amount of data in a group, and `sliding_val` represents the sliding interval.

[0013] As a preferred option, the calculation process of the grouping window function in orderedAggregator is as follows:

[0014] S1. The orderedAggregator retrieves a row of data and checks if the row is empty:

[0015] ① If the data row is not empty, then proceed to step S2;

[0016] ②If the data row is empty, then proceed to step S9;

[0017] S2. Determine if the temporary row container has been initialized:

[0018] ① If the temporary row container is not initialized, it means that the data row is the first row, and step S3 is executed;

[0019] ②If the temporary line container has been initialized, then proceed to step S4;

[0020] S3. Create a row data storage container (rowContainer) and initialize an iter pointing to the first row of the rowContainer. At the same time, initialize countWindowHelper{countValue:0, windowNum, slidingWindowSize}; where windowNum is count_val of count_window(count_val[,sliding_val]), representing that a window has windowNum rows of data; slidingWindowSize is sliding_val, representing the number of rows to slide each time; countValue represents the data rows recorded in the corresponding window. The next step is to execute step S4.

[0021] S4. Set a groupValue as a consecutive window number starting from 0, add the row to the row data storage container rowContainer, and then proceed to step S5.

[0022] S5. Is countValue equal to windowNum?

[0023] ① If countValue equals windowNum, then increment the window number groupValue by 1, reset countValue to 0, delete the first slidingWindowSize data in the row data storage container rowContainer to achieve the behavior of sliding the window backward, and point iter back to the first row of the row data storage container rowContainer. Then proceed to step S6.

[0024] ②If countValue is not equal to windowNum, then proceed to step S6;

[0025] S6. Take a row of data row1 from the row data storage container rowContainer, move iter one position to the right, reuse row to assign row1 to row, copy{row,row1}, and then proceed to step S7.

[0026] S7. Set the auxiliary group window function column in row to groupValue, that is, modify the value of the group function column row[groupWindowCol] = groupValue, and then proceed to step S8.

[0027] S8. The orderedAggregator performs grouping and calculations based on the grouping column after group by. The next step is to jump to step S1.

[0028] S9. The original data has been processed. The remaining data in the row data storage container (rowContainer) is then processed, specifically whether the row data storage container (rowContainer) is empty or whether countValue equals windowNum.

[0029] ①If countValue is less than windowNum, then proceed to step S6;

[0030] ②If countValue equals windowNum, delete the first slidingWindowSize data in the row data storage container rowContainer. Stop deleting when the row data storage container rowContainer is empty; proceed to step S10 next.

[0031] S10. Determine if the row data storage container (rowContainer) is empty:

[0032] ①If the remaining data in the row data storage container rowContainer is not empty, then increment groupValue by 1, set contValue to 0, execute the first row of the container iter, and then jump to step S6.

[0033] ②If the row data storage container rowContainer is empty, the process ends.

[0034] Ideally, when countValue equals windowNum, it means that a window is full and the next window needs to be switched.

[0035] Preferably, when sliding_val is 0 in count_window(count_val[,sliding_val]), it indicates that the windows have no overlapping parts. The specific calculation process of the grouping window function in orderedAggregator is as follows:

[0036] (1) The orderedAggregator retrieves a row of data and checks if the row is empty:

[0037] ①If row is not empty, then proceed to step (2);

[0038] ②If row is empty, the process ends;

[0039] (2) Determine if the result of the modulo operation using groupValue%count_val is equal to 0:

[0040] ① If groupValue%count_val equals 0, it means that the group needs to be switched. Increment groupValue by 1 and proceed to step (3) next.

[0041] ②If groupValue%count_val is not equal to 0, then execute step (3);

[0042] (3) Modify the value of the grouping function column to groupValue;

[0043] (4) Execute the subsequent process of orderedAggretor, perform group calculations, and jump to step (1).

[0044] An electronic device includes: a memory and at least one processor;

[0045] The memory contains computer programs;

[0046] The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the grouped data processing method based on the distributed database KaiwuDB as described above.

[0047] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the grouped data processing method based on the distributed database KaiwuDB as described above.

[0048] `count_window(count_val[,sliding_val])` is a counting window function that divides the window according to the number of rows `count_val`. By default, the data is sorted by the timestamp column. When the cumulative number of rows reaches `count_val`, the data from row 1 to `count_val` is grouped together for aggregation calculation. This process is repeated until all data is read. Because `count_window` is a grouping function that reuses the grouping functionality of `group by`, it needs to be used in conjunction with `group by` to achieve the purpose of grouping. `sliding_val` is the sliding interval. When the current group is [i, i + count_val], the data in the next interval is [i + sliding_val, i + count_val + sliding_val]. Therefore, the `count_window` grouping function allows data overlap.

[0049] The grouped data processing method, device, and medium based on the distributed database KaiwuDB of the present invention have the following advantages:

[0050] (i) This invention can implement the window grouping function count_window in a distributed database, and can similarly implement other grouping window functions such as time_window / state_window / session_window / event_window;

[0051] (ii) This invention adds a grouping window function to a distributed database to increase the database's ability to manipulate data. Attached Figure Description

[0052] The invention will be further described below with reference to the accompanying drawings.

[0053] Appendix Figure 1 This is a flowchart illustrating a grouped data processing method based on the distributed database KaiwuDB.

[0054] Appendix Figure 2 This is a diagram illustrating the modification of grouping column data in orderedAggregator;

[0055] Appendix Figure 3 The flowchart shows the calculation process of the grouped window function count_window;

[0056] Appendix Figure 4 The flowchart shows the calculation process of the grouped window function count_window when the sliding size sliding_val is equal to 0. Detailed Implementation

[0057] The following detailed description of the grouped data processing method, device, and medium based on the distributed database KaiwuDB, with reference to the accompanying drawings and specific embodiments, is provided in the specification.

[0058] Example 1:

[0059] This embodiment provides a grouped data processing method based on the distributed database KaiwuDB. This method utilizes the execution flow of the orderedAggregator, nesting a grouping window function within the orderedAggregator. Before the orderedAggregator performs actual grouping, the values ​​of the grouping columns in the data rows are replaced with a window number. After the grouping column values ​​in the row data are replaced with the window number, the grouping characteristics of the orderedAggregator are used for grouping. Then, the data is sorted according to the time column in each node of the table data distribution. The sorted data is then heap-sorted on the gateway node. Each sorted row is sent to the upper-level orderedAggregator. The orderedAggregator calculates the grouping window function on the input ordered data, as shown in the attached figure. Figure 1 As shown, the table data is distributed across three nodes: node1, node2, and node3. gRPC represents a method for data transmission between nodes. `group by` is an SQL statement that groups the result set based on one or more columns. `orderedAggregator` represents an implementation method for aggregation calculations based on grouped data, requiring the input data to be ordered. The `count_window(count_val[,sliding_val])` function groups data according to a fixed number of rows. This function has two parameters: `count_val` and `sliding_val`. `count_val` is the maximum number of rows in a group, and `sliding_val` is the sliding range. For example, with ten rows of data [1,2,3,4,5,6,7,8,9,10], `count_window(4,2)` actually groups the data into [1,2,3,4], [3,4,5,6], [5,6,7,8], and [9,10]. As shown in the example above, when the final data is less than `count_val`, the number of rows in each group will be less than `count_val`. An example of the SQL statement is shown below:

[0060] SQL

[0061] --Group the array into groups of 4 rows each. After aggregation, slide the data forward 2 rows and continue grouping into 4-row groups. `ts` is the column name in table `t1`, storing the timestamp for each row.

[0062] select first(ts)as first,count(*)from tsdb.t1 group by count_window(4,2).

[0063] The grouping using the orderedAggregator grouping feature in this embodiment is as follows:

[0064] Each row's grouping window column value is set to `groupValue`, starting from 0. Each time a new window is accessed, `groupValue` is incremented by 1, ensuring that the grouping column values ​​for different groups processed by `orderedAggregator` are different, thus achieving grouping. (See attached image) Figure 2 As shown, when there are 8 rows of data, the result of assigning `count_window(4,0)` is as follows. Figure 2 As shown, the grouping column of rows in the same group is assigned the same groupValue.

[0065] In this embodiment, the grouping window function `count_window(count_val[,sliding_val])` divides the window according to the number of data rows `count_val`. By default, the data is sorted by the timestamp column. When the cumulative number of data rows reaches `count_val`, the data from row 1 to row `count_val` is grouped together for aggregation calculation. This process is repeated until all data is read. Here, `count_val` represents the maximum amount of data in a group, and `sliding_val` represents the sliding interval.

[0066] As attached Figure 3 As shown, the specific calculation process of the grouping window function in the orderedAggregator in this embodiment is as follows:

[0067] S1. The orderedAggregator retrieves a row of data and checks if the row is empty:

[0068] ① If the data row is not empty, then proceed to step S2;

[0069] ②If the data row is empty, then proceed to step S9;

[0070] S2. Determine if the temporary row container has been initialized:

[0071] ① If the temporary row container is not initialized, it means that the data row is the first row, and step S3 is executed;

[0072] ②If the temporary line container has been initialized, then proceed to step S4;

[0073] S3. Create a row data storage container (rowContainer) and initialize an iter pointing to the first row of the rowContainer. At the same time, initialize countWindowHelper{countValue:0, windowNum, slidingWindowSize}; where windowNum is count_val of count_window(count_val[,sliding_val]), representing that a window has windowNum rows of data; slidingWindowSize is sliding_val, representing the number of rows to slide each time; countValue represents the data rows recorded in the corresponding window. The next step is to execute step S4.

[0074] S4. Set a groupValue as a consecutive window number starting from 0, add the row to the row data storage container rowContainer, and then proceed to step S5.

[0075] S5. Is countValue equal to windowNum?

[0076] ① If countValue equals windowNum, then increment the window number groupValue by 1, reset countValue to 0, delete the first slidingWindowSize data in the row data storage container rowContainer to achieve the behavior of sliding the window backward, and point iter back to the first row of the row data storage container rowContainer. Then proceed to step S6. When countValue equals windowNum, it means that a window is full and it is necessary to switch to the next window.

[0077] ②If countValue is not equal to windowNum, then proceed to step S6;

[0078] S6. Take a row of data row1 from the row data storage container rowContainer, move iter one position to the right, reuse row to assign row1 to row, copy{row,row1}, and then proceed to step S7.

[0079] S7. Set the auxiliary group window function column in row to groupValue, that is, modify the value of the group function column row[groupWindowCol] = groupValue, and then proceed to step S8.

[0080] S8. The orderedAggregator performs grouping and calculations based on the grouping column after group by. The next step is to jump to step S1.

[0081] S9. The original data has been processed. The remaining data in the row data storage container (rowContainer) is then processed, specifically whether the row data storage container (rowContainer) is empty or whether countValue equals windowNum.

[0082] ①If countValue is less than windowNum, then proceed to step S6;

[0083] ②If countValue equals windowNum, delete the first slidingWindowSize data in the row data storage container rowContainer. Stop deleting when the row data storage container rowContainer is empty; proceed to step S10 next.

[0084] S10. Determine if the row data storage container (rowContainer) is empty:

[0085] ①If the remaining data in the row data storage container rowContainer is not empty, then increment groupValue by 1, set contValue to 0, execute the first row of the container iter, and then jump to step S6.

[0086] ②If the row data storage container rowContainer is empty, the process ends.

[0087] As attached Figure 4 As shown, when sliding_val is 0 in count_window(count_val[,sliding_val]), it indicates that the windows have no overlapping parts. The specific calculation process of the grouping window function in orderedAggregator is as follows:

[0088] (1) The orderedAggregator retrieves a row of data and checks if the row is empty:

[0089] ①If row is not empty, then proceed to step (2);

[0090] ②If row is empty, the process ends;

[0091] (2) Determine if the result of the modulo operation using groupValue%count_val is equal to 0:

[0092] ① If groupValue%count_val equals 0, it means that the group needs to be switched. Increment groupValue by 1 and proceed to step (3) next.

[0093] ②If groupValue%count_val is not equal to 0, then execute step (3);

[0094] (3) Modify the value of the grouping function column to groupValue;

[0095] (4) Execute the subsequent process of orderedAggretor, perform group calculations, and jump to step (1).

[0096] Example 2:

[0097] This embodiment also provides an electronic device, including: a memory and a processor;

[0098] The memory stores the instructions executed by the computer.

[0099] The processor executes computer execution instructions stored in the memory, causing the processor to perform the grouped data processing method based on the distributed database KaiwuDB in any embodiment of the present invention.

[0100] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0101] Memory is used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory devices, or other volatile solid-state storage devices.

[0102] Example 3:

[0103] This embodiment also provides a computer-readable storage medium storing multiple instructions, which are loaded by a processor to cause the processor to execute the grouped data processing method based on the distributed database KaiwuDB according to any embodiment of the present invention. Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0104] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0105] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0106] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0107] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A grouped data processing method based on the distributed database KaiwuDB, characterized in that, This method utilizes the execution flow of orderedAggregator, nesting the grouping window function within orderedAggregator. Before orderedAggregator performs actual grouping, the value of the grouping window column for each row is assigned to groupValue, which starts from 0. Each time a new window is entered, groupValue is incremented by 1, ensuring that the grouping column values ​​for different groups processed by orderedAggregator are different, thus achieving grouping. When the value of the grouping column in the row data is replaced by the window number, the grouping feature of orderedAggregator is used for grouping. The specific grouping process is as follows: the data is sorted according to the time column in each node of the table data distribution. The sorted data is then heap-sorted on the gateway node. Each row can be sent to the upper-level orderedAggregator. orderedAggregator calculates the grouping window function on the input ordered data.

2. The grouped data processing method based on the distributed database KaiwuDB according to claim 1, characterized in that, The grouping window function `count_window(count_val[,sliding_val])` divides the window according to the number of data rows `count_val`. By default, the data is sorted by the timestamp column. When the cumulative number of data rows reaches `count_val`, the data from row 1 to `count_val` is grouped together and aggregated. This process is repeated until all data is read. Here, `count_val` represents the maximum number of data in a group, and `sliding_val` represents the sliding interval.

3. The grouped data processing method based on the distributed database KaiwuDB according to claim 1, characterized in that, The specific computation process of the grouping window function in orderedAggregator is as follows: S1. The orderedAggregator retrieves a row of data and checks if the row is empty: ① If the data row is not empty, then proceed to step S2; ②If the data row is empty, then proceed to step S9; S2. Determine if the temporary row container has been initialized: ① If the temporary row container is not initialized, it means that the data row is the first row, and step S3 is executed; ②If the temporary line container has been initialized, then proceed to step S4; S3. Create a row data storage container (rowContainer) and initialize an iter pointing to the first row of the rowContainer. At the same time, initialize countWindowHelper{countValue:0, windowNum,slidingWindowSize}. Here, windowNum is count_val of count_window(count_val[,sliding_val]), which represents that a window has windowNum rows of data; slidingWindowSize is sliding_val, which represents the number of rows to slide each time; countValue represents the data rows recorded in the corresponding window. The next step is to execute step S4. S4. Set a groupValue as a consecutive window number starting from 0, add the row to the row data storage container rowContainer, and then proceed to step S5. S5. Is countValue equal to windowNum? ①If countValue equals windowNum, then increment the window number groupValue by 1, reset countValue to 0, delete the first slidingWindowSize data in the row data storage container rowContainer to achieve the behavior of sliding the window backward, and point iter back to the first row of the row data storage container rowContainer. Then proceed to step S6. ②If countValue is not equal to windowNum, then proceed to step S6; S6. Take a row of data row1 from the row data storage container rowContainer, move iter one position to the right, reuse row to assign row1 to row, copy{row,row1}, and then proceed to step S7. S7. Set the auxiliary group window function column in row to groupValue, that is, modify the value of the group function column row[groupWindowCol] = groupValue, and then proceed to step S8. S8. The orderedAggregator performs grouping and calculations based on the grouping column after group by. The next step is to jump to step S1. S9. The original data has been processed. The remaining data in the row data storage container (rowContainer) is then processed, specifically whether the row data storage container (rowContainer) is empty or whether the countValue is equal to windowNum. ①If countValue is less than windowNum, then proceed to step S6; ②If countValue equals windowNum, delete the first slidingWindowSize data in the row data storage container rowContainer. Stop deleting when the row data storage container rowContainer is empty; proceed to step S10 next. S10. Determine if the row data storage container (rowContainer) is empty: ①If the remaining data in the row data storage container rowContainer is not empty, then increment groupValue by 1, set contValue to 0, execute the first row of the container iter, and then jump to step S6. ②If the row data storage container rowContainer is empty, the process ends.

4. The grouped data processing method based on the distributed database KaiwuDB according to claim 3, characterized in that, When countValue equals windowNum, it means that a window is full and you need to switch to the next window.

5. The grouped data processing method based on the distributed database KaiwuDB according to claim 1, characterized in that, When sliding_val is 0 in count_window(count_val[,sliding_val]), it means that the windows have no overlapping parts. The specific calculation process of the grouping window function in orderedAggregator is as follows: (1) The orderedAggregator retrieves a row of data and checks if the row is empty: ①If row is not empty, then proceed to step (2); ②If row is empty, the process ends; (2) Determine if the result of the modulo operation using groupValue%count_val is equal to 0: ① If groupValue%count_val equals 0, it means that the group needs to be switched. Increment groupValue by 1 and proceed to step (3). ②If groupValue%count_val is not equal to 0, then execute step (3); (3) Modify the value of the grouping function column to groupValue; (4) Execute the subsequent process of orderedAggretor, perform group calculations, and jump to step (1).

6. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the grouped data processing method based on the distributed database KaiwuDB as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the grouped data processing method based on the distributed database KaiwuDB as described in any one of claims 1 to 5.

Images