Distributed database KaiwuDB-based grouped data processing method, equipment and medium

Abstract

The invention discloses a grouped data processing method and equipment based on a distributed database KaiwuDB and a medium, belongs to the technical field of databases, and aims to solve the technical problem of how to increase the data processing capacity of a database and improve the data processing efficiency. According to the technical scheme, a grouped window function is nested in an orderdAggregate by utilizing an execution process of orderdAggregate r, and the data processing efficiency is improved; the method comprises the following steps of: grouping data in a data row by using a window number, replacing a value of a grouping column of the data row by using a window number before executing actual grouping by using an orderdAggregate, performing grouping by using an orderdAggregate grouping characteristic after the value of the grouping column in the row data is replaced by the window number, sorting according to a time column in each node where table data is distributed, and performing heap sorting on the sorted data at a gateway node, a row of data can be sent to an upper layer ordedAggregate by each row of data, and the ordedAggregate is used for carrying out grouping window function calculation on the input ordered data.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a group data processing method, device and medium based on the distributed database KaiwuDB. Background Art

[0002] A distributed database is a cluster of multiple database nodes, where data can be stored. KaiwuDB is a distributed, multi-mode database designed for AIoT (Artificial Intelligence of Things) scenarios. KaiwuDB's data processing methods include a distributed architecture and horizontal expansion, a multi-mode fusion engine, time series data engine optimization, native AI integration, efficient compression and hot and cold tiered storage, transaction processing and consistency assurance, and out-of-order data processing and deduplication. However, KaiwuDB's current data processing methods have the following flaws:

[0003] ① Relatively high transaction latency: Although transaction latency has been reduced through optimization strategies, the decentralized design may still lead to high transaction rollback rates and rollback costs in high-concurrency and high-conflict scenarios.

[0004] ② High learning cost: Features such as multi-mode fusion and native AI integration increase the complexity of the system and require high technical capabilities of development and operation and maintenance personnel.

[0005] ③High resource consumption: The design of storing locks and data together may cause each write operation to generate two or three times the write pressure, which may affect performance in an environment with limited resources.

[0006] ④ Adaptability to specific scenarios: Although time series data processing has been optimized, some advanced features may not be able to fully demonstrate their advantages in non-AIoT scenarios or low-concurrency scenarios.

[0007] Therefore, how to increase the database's data processing capabilities and improve data processing efficiency is a technical problem that needs to be solved urgently. Summary of the Invention

[0008] The technical task of the present invention is to provide a group data processing method, device and medium based on the distributed database KaiwuDB to solve the problem of how to increase the database's data processing capabilities and improve data processing efficiency.

[0009] The technical task of the present invention is achieved in the following manner: a grouped data processing method based on the distributed database KaiwuDB, which utilizes the execution process of orderedAggregator, nests the grouping window function in orderedAggregator, and replaces the value of the grouping column of the data row with a window number before the orderedAggregator performs actual grouping. After the value of the grouping column in the row data is replaced by the window number, the orderedAggregator grouping feature is used to group the data, and then sort the data according to the time column in each node where the table data is distributed. The sorted data is heap sorted at the gateway node, and each row can send a row of data to the upper-level orderedAggregator. The orderedAggregator performs grouping window function calculation on the input ordered data.

[0010] As a preference, the grouping characteristics of orderedAggregator are used as follows:

[0011] The group window column amplitude of each row is groupValue. GroupValue starts from 0. Every time a new window is reached, groupValue is increased by 1 to ensure that the group column values ​​of different groups processed by orderedAggregator are different, thereby achieving grouping.

[0012] As a preferred method, the grouping window function count_window(count_val[,sliding_val]) divides the window according to the number of data rows count_val, and sorts the data by the timestamp column by default. When the cumulative number of data rows reaches count_val, the data from row 1 to row count_val are grouped together for aggregation calculation. The calculation loop continues to group all data by count_val until all data is read. Here, count_val represents the maximum amount of data in a group; sliding_val represents the sliding interval.

[0013] As a preference, the group window function calculation process in orderedAggregator is as follows:

[0014] S1. OrderedAggregator takes a row of data and determines whether the row is empty:

[0015] ① If the data row is not empty, execute step S2;

[0016] ② If the data row is empty, execute step S9;

[0017] S2. Determine whether the temporary row container is initialized:

[0018] ① If the temporary row container is not initialized, it means that the data row is the first row, and step S3 is executed;

[0019] ② If the temporary row container has been initialized, execute step S4;

[0020] S3. Create a row data storage container rowContainer and initialize an iter to point to the first row of rowContainer. At the same time, initialize countWindowHelper{countValue:0,windowNum,slidingWindowSize}; where windowNum is count_val in count_window(count_val[,sliding_val]), indicating that a window has windowNum rows of data; slidingWindowSize is sliding_val, indicating the number of rows to slide each time; countValue represents the data rows recorded in the corresponding window. Next, execute step S4;

[0021] S4. Set a groupValue as a continuous window number starting from 0, add row to the row data storage container rowContainer, and then execute step S5;

[0022] S5. Is countValue equal to windowNum?

[0023] ① If countValue is equal to windowNum, then add 1 to the window number groupValue, reset countVa lue to 0, delete the first slidingWindowSize data in the row data storage container rowContainer to achieve the behavior of sliding the window backward, and point iter to the first row of the row data storage container rowContainer again. Next, execute step S6;

[0024] ② If countValue is not equal to windowNum, execute step S6;

[0025] S6. Take a row of data row1 from the row data storage container rowContainer, move iter one position backward, reuse row to assign row1 to row, copy{row,row1}, and then execute step S7.

[0026] S7, the group window function column in row is auxiliary to groupValue, that is, the value of the group function column is modified row[groupWindowCol] = groupValue, and the next step is to execute step S8;

[0027] S8, orderedAggregator performs group calculation according to the grouping column after group by, and then jumps to step S1;

[0028] S9. The original data has been processed, and the remaining data in the row data storage container rowContainer is processed, that is, whether the row data storage container rowContainer is empty or whether countVa lue is equal to windowNum:

[0029] ① If countValue is less than windowNum, jump to step S6;

[0030] ② If countValue is equal to windowNum, delete the first slidingWindowSize data in the row data storage container rowContainer. When the row data storage container rowContainer is empty, stop deleting; the next step is to execute step S10;

[0031] S10. Determine whether the row data storage container rowContainer is empty:

[0032] ① If the remaining data in the row data storage container rowContainer is not empty, then groupValue is increased by 1, contValue is set to 0, iter is executed on the first row of the container, and the next step jumps to step S6;

[0033] ② If the remaining data in the row data storage container rowContainer is empty, the process ends.

[0034] Preferably, when countValue is equal to windowNum, it indicates that a window is full and the next window needs to be switched.

[0035] As a preference, when sliding_val in count_window(count_val[,sliding_val]) is 0, it indicates that there is no overlapping of windows. The calculation process of the grouping window function in orderedAggregator is as follows:

[0036] (1) OrderedAggregator takes a row of data and determines whether the row is empty:

[0037] ①If row is not empty, execute step (2);

[0038] ②If row is empty, the process ends;

[0039] (2) Determine whether the result of the modulo operation using groupValue%count_val is equal to 0:

[0040] ① If groupValue%count_val is equal to 0, it means that the group needs to be switched, groupValue is increased by 1, and the next step is to execute step (3);

[0041] ② If groupValue%count_val is not equal to 0, execute step (3);

[0042] (3) Modify the value of the grouping function column to groupValue;

[0043] (4) Execute the subsequent process of orderedAggretor, perform group calculation, and jump to step (1).

[0044] An electronic device comprising: a memory and at least one processor;

[0045] Wherein, the memory stores a computer program;

[0046] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the above-mentioned group data processing method based on the distributed database KaiwuDB.

[0047] A computer-readable storage medium stores a computer program, which can be executed by a processor to implement the above-mentioned group data processing method based on the distributed database KaiwuDB.

[0048] Among them, count_window(count_val[,sliding_val]) is a counting window function. It divides the window according to the number of data rows (count_val) and sorts the data by the timestamp column by default. When the cumulative number of data rows reaches count_val, the data from rows 1 to count_val are grouped together for aggregation calculation. The calculation loop continues until all data is read. Because count_window is a grouping function and the grouping function reuses the group by statement, it must be used in conjunction with group by to achieve the grouping purpose. sliding_val is a sliding interval. When the current group is [i,i+count_val], the data in the next interval is [i+sliding_val,i+count_val+sliding_val]. Therefore, the count_window grouping function allows data overlap.

[0049] The group data processing method, device, and medium based on the distributed database KaiwuDB of the present invention have the following advantages:

[0050] (1) The present invention can implement the window grouping function count_window in a distributed database, and can similarly implement other grouping window functions such as time_window / state_window / session_window / event_window;

[0051] (2) The present invention adds a grouping window function to the distributed database to increase the database's ability to operate on data. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The present invention will be further described below with reference to the accompanying drawings.

[0053] Attachment Figure 1 A flowchart of a group data processing method based on the distributed database KaiwuDB;

[0054] Attachment Figure 2 Schematic diagram for modifying group column data in orderedAggregator;

[0055] Attachment Figure 3 This is a flowchart of the calculation process of the group window function count_window;

[0056] Attachment Figure 4 This is a flowchart of the calculation process of the grouping window function count_window when the sliding size sliding_val is equal to 0. DETAILED DESCRIPTION

[0057] The group data processing method, device and medium based on the distributed database KaiwuDB of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] Example 1:

[0059] This embodiment provides a grouped data processing method based on the distributed database KaiwuDB. The method utilizes the execution process of orderedAggregator, nests the grouping window function in orderedAggregator, and replaces the value of the grouping column of the data row with a window number before the orderedAggregator performs actual grouping. After the value of the grouping column in the row data is replaced by the window number, the grouping is performed using the grouping feature of orderedAggregator. Then, the data is sorted according to the time column in each node where the table data is distributed. The sorted data is heap-sorted at the gateway node. Each row can send a row of data to the upper-level orderedAggregator. The orderedAggregator performs grouped window function calculation on the input ordered data, as shown in the attached figure. Figure 1 As shown, the table data is distributed among three nodes: node1, node2, and node3; gRPC represents a means for each node to transmit data; group by is an SQL statement that groups the result set according to one or more columns; orderedAggregator represents an implementation method for performing aggregate calculations on data in a grouped manner, requiring the input data to be ordered; the count_window(count_val[,sliding_val]) function represents grouping data according to a fixed number of rows. This function has two parameters: count_val and sliding_val. Count_val is the maximum amount of data in a group, and sliding_val is the sliding interval. Taking the ten rows of data [1,2,3,4,5,6,7,8,9,10] as an example, count_window(4,2) actually groups them into [1,2,3,4], [3,4,5,6], [5,6,7,8], and [9,10]. As shown in the above example, when the last data is less than count_val, the data in the group will be less than count_val. The SQL statement example is as follows:

[0060] SQL

[0061] --Group the array into groups of 4 rows. After aggregation, slide the data back 2 rows and continue grouping by 4 rows. ts is the name of the column in table t1, storing the timestamp of each row.

[0062] select first(ts)as first,count(*)from tsdb.t1 group by count_window(4,2).

[0063] In this embodiment, the grouping using the orderedAggregator grouping feature is specifically as follows:

[0064] The group window column amplitude of each row is groupValue, groupValue starts from 0, and each time a new window is reached, groupValue is increased by 1 to ensure that the group column values ​​of different groups processed by orderedAggregator are different, thereby achieving grouping. Figure 2 As shown, when there are 8 rows of data, count_window(4,0), the result after assignment is as shown in the attached Figure 2 As shown, the grouping columns of rows in the same group are assigned the same groupValue.

[0065] The grouping window function count_window(count_val[,sliding_val]) in this embodiment divides the window according to the number of data rows count_val, and sorts the data by the timestamp column by default. When the cumulative number of data rows reaches count_val, the data from row 1 to row count_val are grouped together for aggregation calculation. The calculation loop continues, grouping all data by count_val until all data is read. Here, count_val represents the maximum amount of data in a group; sliding_val represents the sliding interval.

[0066] As attached Figure 3 As shown, the calculation process of the group window function in the orderedAggregator in this embodiment is as follows:

[0067] S1. OrderedAggregator takes a row of data and determines whether the row is empty:

[0068] ① If the data row is not empty, execute step S2;

[0069] ② If the data row is empty, execute step S9;

[0070] S2. Determine whether the temporary row container is initialized:

[0071] ① If the temporary row container is not initialized, it means that the data row is the first row, and step S3 is executed;

[0072] ② If the temporary row container has been initialized, execute step S4;

[0073] S3. Create a row data storage container rowContainer and initialize an iter to point to the first row of rowContainer. At the same time, initialize countWindowHelper{countValue:0,windowNum,slidingWindowSize}; where windowNum is count_val in count_window(count_val[,sliding_val]), indicating that a window has windowNum rows of data; slidingWindowSize is sliding_val, indicating the number of rows to slide each time; countValue represents the data rows recorded in the corresponding window. Next, execute step S4;

[0074] S4. Set a groupValue as a continuous window number starting from 0, add row to the row data storage container rowContainer, and then execute step S5;

[0075] S5. Is countValue equal to windowNum?

[0076] ① If countValue is equal to windowNum, then the window number groupValue is increased by 1, countVa lue is reset to 0, the first slidingWindowSize data in the row data storage container rowContainer is deleted to achieve the behavior of sliding the window backward, and iter is pointed to the first row of the row data storage container rowContainer again. Next, step S6 is executed; among them, when countValue is equal to windowNum, it means that one window is full and it is necessary to switch to the next window;

[0077] ② If countValue is not equal to windowNum, execute step S6;

[0078] S6. Take a row of data row1 from the row data storage container rowContainer, move iter one position backward, reuse row to assign row1 to row, copy{row,row1}, and then execute step S7.

[0079] S7, the group window function column in row is auxiliary to groupValue, that is, the value of the group function column is modified row[groupWindowCol] = groupValue, and the next step is to execute step S8;

[0080] S8, orderedAggregator performs group calculation according to the grouping column after group by, and then jumps to step S1;

[0081] S9. The original data has been processed, and the remaining data in the row data storage container rowContainer is processed, that is, whether the row data storage container rowContainer is empty or whether countVa lue is equal to windowNum:

[0082] ① If countValue is less than windowNum, jump to step S6;

[0083] ② If countValue is equal to windowNum, delete the first slidingWindowSize data in the row data storage container rowContainer. When the row data storage container rowContainer is empty, stop deleting; the next step is to execute step S10;

[0084] S10. Determine whether the row data storage container rowContainer is empty:

[0085] ① If the remaining data in the row data storage container rowContainer is not empty, then groupValue is increased by 1, contValue is set to 0, iter is executed on the first row of the container, and the next step jumps to step S6;

[0086] ② If the remaining data in the row data storage container rowContainer is empty, the process ends.

[0087] As attached Figure 4 As shown in the figure, when sliding_val in count_window(count_val[,sliding_val]) is 0, it means that there is no overlapping part of the window. The calculation process of the group window function in orderedAggregator is as follows:

[0088] (1) OrderedAggregator takes a row of data and determines whether the row is empty:

[0089] ①If row is not empty, execute step (2);

[0090] ②If row is empty, the process ends;

[0091] (2) Determine whether the result of the modulo operation using groupValue%count_val is equal to 0:

[0092] ① If groupValue%count_val is equal to 0, it means that the group needs to be switched, groupValue is increased by 1, and the next step is to execute step (3);

[0093] ② If groupValue%count_val is not equal to 0, execute step (3);

[0094] (3) Modify the value of the grouping function column to groupValue;

[0095] (4) Execute the subsequent process of orderedAggretor, perform group calculation, and jump to step (1).

[0096] Example 2:

[0097] This embodiment also provides an electronic device, including: a memory and a processor;

[0098] wherein the memory stores computer-executable instructions;

[0099] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the group data processing method based on the distributed database KaiwuDB in any embodiment of the present invention.

[0100] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0101] The memory can be used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, the memory can also include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state memory devices.

[0102] Example 3:

[0103] This embodiment further provides a computer-readable storage medium storing a plurality of instructions, which are loaded by a processor to cause the processor to execute the packet data processing method based on the distributed database KaiwuDB according to any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, wherein the storage medium stores software program code that implements the functions of any of the above-described embodiments, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0104] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.

[0105] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RYMs, DVD-RWs, DVD+RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer via a communications network.

[0106] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0107] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program code, the CPU installed on the expansion board or expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing grouped data based on the distributed database KaiwuDB, characterized in that: This method uses the execution process of orderedAggregator to nest the grouping window function in orderedA ggregator. Before orderedAggregator performs actual grouping, the value of the grouping column of the data row is replaced with a window number. After the value of the grouping column in the row data is replaced by the window number, the grouping feature of orderedA ggregator is used to group the data. Then, the data is sorted according to the time column in each node where the table data is distributed. The sorted data is heap-sorted at the gateway node. Each row can be sent to the upper-level orderedA ggregator. OrderedAggregator calculates the grouping window function on the input ordered data.

2. The method for processing grouped data based on the distributed database KaiwuDB according to claim 1, characterized in that: The grouping characteristics of orderedAggregator are as follows: The group window column amplitude of each row is groupValue. GroupValue starts from 0. Every time a new window is reached, groupValue is increased by 1 to ensure that the group column values ​​of different groups processed by orderedAggregator are different, thereby achieving grouping.

3. The method for processing grouped data based on the distributed database KaiwuDB according to claim 1, characterized in that: The grouping window function count_window(count_val[,sliding_val]) divides the window into rows by count_val and sorts the data by the timestamp column by default. When the cumulative number of rows reaches count_val, the data from row 1 to row count_val are grouped together for aggregation calculation. This loop continues until all data is read, grouping them by count_val. Count_val represents the maximum amount of data in a group, and sliding_val represents the sliding interval.

4. The method for processing grouped data based on the distributed database KaiwuDB according to claim 1, characterized in that: The calculation process of the group window function in orderedAggregator is as follows: S1. OrderedAggregator takes a row of data and determines whether the row is empty: ① If the data row is not empty, execute step S2; ② If the data row is empty, execute step S9; S2. Determine whether the temporary row container is initialized: ① If the temporary row container is not initialized, it means that the data row is the first row, and step S3 is executed; ② If the temporary row container has been initialized, execute step S4; S3. Create a row data storage container rowContainer and initialize an iter to point to the first row of rowContainer. At the same time, initialize countWindowHelper{countValue:0,windowNum,slidingWindowSize}; where windowNum is count_val in count_window(count_val[,sliding_val]), indicating that a window has windowNum rows of data; slidingWindowSize is sliding_val, indicating the number of rows to slide each time; countValue represents the data rows recorded in the corresponding window. Next, execute step S4; S4. Set a groupValue as a continuous window number starting from 0, add row to the row data storage container rowContainer, and then execute step S5; S5. Is countValue equal to windowNum? ① If countValue is equal to windowNum, then add 1 to the window number groupValue, reset countVa lue to 0, delete the first slidingWindowSize data in the row data storage container rowContainer to achieve the behavior of sliding the window backward, and point iter to the first row of the row data storage container rowContainer again. Next, execute step S6; ② If countValue is not equal to windowNum, execute step S6; S6. Take a row of data row1 from the row data storage container rowContainer, move iter one position backward, reuse row to assign row1 to row, copy{row,row1}, and then execute step S7. S7, the group window function column in row is auxiliary to groupValue, that is, the value of the group function column is modified row[groupWindowCol] = groupValue, and the next step is to execute step S8; S8, orderedAggregator performs group calculation according to the grouping column after group by, and then jumps to step S1; S9. The original data has been processed, and the remaining data in the row data storage container rowContainer is processed, that is, whether the row data storage container rowContainer is empty or whether countVa lue is equal to windowNum: ① If countValue is less than windowNum, jump to step S6; ② If countValue is equal to windowNum, delete the first slidingWindowSize data in the row data storage container rowContainer. When the row data storage container rowContainer is empty, stop deleting; the next step is to execute step S10; S10. Determine whether the row data storage container rowContainer is empty: ① If the remaining data in the row data storage container rowContainer is not empty, then groupValue is increased by 1, contValue is set to 0, iter is executed on the first row of the container, and the next step jumps to step S6; ② If the remaining data in the row data storage container rowContainer is empty, the process ends.

5. The method for processing grouped data based on the distributed database KaiwuDB according to claim 4, characterized in that: When countValue is equal to windowNum, it means that one window is full and the next window needs to be switched.

6. The method for processing grouped data based on the distributed database KaiwuDB according to claim 1, characterized in that: When sliding_val in count_window(count_val[,sliding_val]) is 0, it means that there is no overlapping window. The calculation process of the group window function in orderedAggregator is as follows: (1) OrderedAggregator takes a row of data and determines whether the row is empty: ①If row is not empty, execute step (2); ②If row is empty, the process ends; (2) Determine whether the result of the modulo operation using groupValue%count_val is equal to 0: ① If groupValue%count_val is equal to 0, it means that the group needs to be switched, groupValue is increased by 1, and the next step is to execute step (3); ② If groupValue%count_val is not equal to 0, execute step (3); (3) Modify the value of the grouping function column to groupValue; (4) Execute the subsequent process of orderedAggretor, perform group calculation, and jump to step (1).

7. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the group data processing method based on the distributed database KaiwuDB according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the group data processing method based on the distributed database KaiwuDB according to any one of claims 1 to 6.

Images