Data processing method, device, computing equipment and storage medium

Abstract

The present application provides a data processing method, apparatus, computing device, and storage medium. The present application determines the window number of each row of the data to be processed by using the window column data of each row, and compresses and stores the window column data according to the window number of each row, thereby saving memory. Furthermore, based on the window number of each row, index jump information for the same window number is stored as a linked list, and linked list headers corresponding to different window numbers are stored at the linked list header, thereby avoiding the workload of full sorting and shortening the time occupied by window division.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of computer technology, and in particular, to a data processing method, apparatus, computing device, and storage medium. Background Art

[0002] In the era of big data, vast quantities of data are generated across all industries. Analyzing this data can further guide production optimization and business direction. Window operators are commonly used data analysis operators that divide data into windows based on their characteristics and then perform further calculations on the data in each window. Currently, using sorting algorithms to divide data into windows is complex and time-consuming. Summary of the Invention

[0003] In view of this, one or more embodiments of this specification provide a data processing method, apparatus, computing device, and medium.

[0004] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0005] According to a first aspect of one or more embodiments of this specification, a data processing method is provided, the method comprising:

[0006] Acquire data to be processed, and determine a window column and a data column in the data to be processed, wherein the window column is used to indicate that the data to be processed is divided into windows according to the data in the column;

[0007] Determining the window number of each row according to the window column data of each row in the data to be processed;

[0008] Compress and store window column data according to the window number of each row;

[0009] According to the window number of each row, the index of each row data column data is stored as a linked list, and the linked list headers corresponding to different window numbers are stored in the linked list header, wherein the index in the linked list is used to indicate the index jump information of the same window number.

[0010] In one embodiment of the present specification, determining the window number of each row according to the window column data of each row in the data to be processed includes:

[0011] Determine a hash value of each row of window column data in the data to be processed;

[0012] The hash value is inserted into a hash table, and the window number is determined based on the position of the hash value in the hash table.

[0013] In one embodiment of the present specification, inserting the hash value into a hash table and determining the window number according to a position of the hash value in the hash table includes:

[0014] If the hash value does not exist in the hash table, determining the window number according to the position of the hash value in the hash table, and establishing a mapping relationship between the window number and the hash value;

[0015] In the case that a hash value already exists in the hash table, the window number corresponding to the hash value is determined according to the mapping relationship.

[0016] In one embodiment of the present specification, storing the index of each row of data column data as a linked list according to the window number of each row, and storing the linked list header corresponding to different window numbers at the linked list header, includes: performing the following operations for each row:

[0017] According to the window number of the row, determine the linked list head corresponding to the window number;

[0018] Storing the index stored in the linked list header into the linked list corresponding to the row;

[0019] The index of the row is stored in the head of the linked list.

[0020] In one embodiment of the present specification, before storing the indexes of the rows and columns of data as a linked list, the method further includes:

[0021] An index column is constructed, in which the index of each row of data column data is stored.

[0022] In one embodiment of the present specification, the method further includes:

[0023] Constructing a window index column according to the index in the linked list header and the index jump information of the linked list indicating the same window number;

[0024] Obtain data column data according to the index in the window index column;

[0025] The data in the same window is obtained according to the window column data corresponding to the data column data and the window number.

[0026] In one embodiment of the present specification, constructing a window index column according to the index in the linked list header and the index jump information of the linked list indicating the same window number includes:

[0027] Obtaining a first index in the linked list header, where the first index corresponds to a first window number;

[0028] According to the row indicated by the first index, obtain the second index in the linked list;

[0029] Performing an index jump in the linked list based on the second index to obtain a third index in the linked list indicating the first window number;

[0030] A window index column corresponding to the first window number is obtained according to the first index, the second index, and the third index.

[0031] According to a second aspect of one or more embodiments of this specification, a data processing device is provided, the device comprising:

[0032] an acquiring unit, configured to acquire data to be processed and determine a window column and a data column in the data to be processed, wherein the window column is used to indicate that the data to be processed is divided into windows according to the data in the column;

[0033] a determining unit, configured to determine a window number of each row according to window column data of each row in the data to be processed;

[0034] A compression storage unit, used for compressing and storing window column data according to the window number of each row;

[0035] The first processing unit is used to store the index of the data column of each row as a linked list according to the window number of each row, and store the linked list headers corresponding to different window numbers in the linked list header, wherein the index in the linked list is used to indicate the index jump information of the same window number.

[0036] In one embodiment of this specification, the determining unit is configured to:

[0037] Determine a hash value of each row of window column data in the data to be processed;

[0038] The hash value is inserted into a hash table, and the window number is determined based on the position of the hash value in the hash table.

[0039] In one embodiment of the present specification, the determining unit, when inserting the hash value into the hash table and determining the window number according to the position of the hash value in the hash table, is configured to:

[0040] If the hash value does not exist in the hash table, determining the window number according to the position of the hash value in the hash table, and establishing a mapping relationship between the window number and the hash value;

[0041] In the case that a hash value already exists in the hash table, the window number corresponding to the hash value is determined according to the mapping relationship.

[0042] In one embodiment of the present specification, the first processing unit is configured to perform the following operations for each row:

[0043] According to the window number of the row, determine the linked list head corresponding to the window number;

[0044] Storing the index stored in the linked list header into the linked list corresponding to the row;

[0045] The index of the row is stored in the head of the linked list.

[0046] In one embodiment of the present specification, before storing the index of each row of data column data as a linked list, the apparatus further includes: a construction unit for constructing an index column, wherein the index of each row of data column data is stored in the index column.

[0047] In one embodiment of the present specification, the apparatus further includes a second processing unit configured to:

[0048] Constructing a window index column according to the index in the linked list header and the index jump information of the linked list indicating the same window number;

[0049] Obtain data column data according to the index in the window index column;

[0050] The data in the same window is obtained according to the window column data corresponding to the data column data and the window number.

[0051] In one embodiment of the present specification, the second processing unit, when constructing the window index column according to the index in the linked list header and the index jump information indicating the same window number in the linked list, is configured to:

[0052] Obtaining a first index in the linked list header, where the first index corresponds to a first window number;

[0053] According to the row indicated by the first index, obtain the second index in the linked list;

[0054] Performing an index jump in the linked list based on the second index to obtain a third index in the linked list indicating the first window number;

[0055] A window index column corresponding to the first window number is obtained according to the first index, the second index, and the third index.

[0056] According to a third aspect of one or more embodiments of this specification, a computing device is provided, the computing device including:

[0057] processor;

[0058] a memory for storing processor-executable instructions;

[0059] The processor implements the operations performed by the data processing method provided in the first aspect and any embodiment of the first aspect by running executable instructions.

[0060] According to the fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is proposed, on which computer instructions are stored. When the instructions are executed by a processor, the operations performed by the data processing method provided in the above-mentioned first aspect and any embodiment of the first aspect are implemented.

[0061] According to the fifth aspect of one or more embodiments of this specification, a computer program product is proposed, including a computer program, which, when executed by a processor, implements the operations performed by the data processing method provided in the above-mentioned first aspect and any embodiment of the first aspect.

[0062] This application determines the window number of each row through the window column data of each row in the data to be processed, and compresses and stores the window column data according to the window number of each row, which can save memory. According to the window number of each row, the index jump information of the same window number is stored as a linked list, and the linked list headers corresponding to different window numbers are stored in the linked list header, avoiding the workload of full sorting and shortening the time occupied by window division. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flowchart of a data processing method provided by an exemplary embodiment.

[0064] Figure 2 This is a schematic diagram of a process for compressing data to be processed provided by an exemplary embodiment.

[0065] Figure 3 This is a schematic diagram of constructing an index linked list and a linked list header for data to be processed, provided by an exemplary embodiment.

[0066] Figure 4 This is a schematic diagram of constructing a first row index linked list and a corresponding linked list header provided by an exemplary embodiment.

[0067] Figure 5 This is a schematic diagram of constructing a second row index linked list and a corresponding linked list header provided by an exemplary embodiment.

[0068] Figure 6 This is a schematic diagram of constructing a third row index linked list and a corresponding linked list header provided by an exemplary embodiment.

[0069] Figure 7 This is a schematic diagram of constructing a fourth row index linked list and a corresponding linked list header provided by an exemplary embodiment.

[0070] Figure 8 This is a schematic diagram of constructing a fifth-row index linked list and a corresponding linked list header provided by an exemplary embodiment.

[0071] Figure 9This is a schematic diagram of a process for constructing an index column provided by an exemplary embodiment.

[0072] Figure 10a This is a flowchart of a process for constructing a W1 window index column, shown as an exemplary embodiment.

[0073] Figure 10b This is a flowchart illustrating a process of constructing a W2 window index column according to an exemplary embodiment.

[0074] Figure 11 It is a block diagram of a data processing device provided by an exemplary embodiment.

[0075] Figure 12 It is a schematic structural diagram of a computing device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0076] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments are not intended to represent all implementations consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of one or more embodiments of this specification, as detailed in the appended claims.

[0077] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than those described in this specification. In addition, a single step described in this specification may be broken down into multiple steps for description in other embodiments, and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0078] In the era of big data, all industries generate vast amounts of data. Data analysis can further guide production optimization and business direction, such as analyzing past annual sales figures or user behavior data. This type of data processing is called analytical business and is handled by specialized analytical databases. Analytical databases often need to process massive amounts of data all at once. For example, they might need to sum all transactions in a year to calculate the annual transaction volume. This means that a single query often consumes significant resources and time. Therefore, performance optimization in analytical databases has a significant impact on user experience. Window operators are a common analytical operator in analytical databases. Using window partitioning within a window operator, data can be divided into different windows. Currently, window partitioning methods require a unified sorting of all data after accumulating it. Due to the high complexity of the sorting algorithm, this often consumes significant query time and can easily become a bottleneck for queries. Consequently, using sorting algorithms to perform window partitioning has low performance, resulting in extended response times and a poor user experience.

[0079] In view of this, the present application provides a data processing method that can be applied to scenarios where data is windowed. Windowing in the present application can be understood as dividing rows in the data according to the data in a specified column, where all rows with the same data in that column are grouped together, and rows with different data in that column are grouped together. The specified column is called a window column.

[0080] The data processing method provided in this application can be executed by a computing device, which can be a server, such as a single server, multiple servers, a server cluster, a cloud computing platform, etc. This application does not limit the type and number of computing devices.

[0081] The following embodiments will introduce the specific implementation process of this application in conjunction with the accompanying drawings.

[0082] See also Figure 1 , Figure 1 1 is a flow chart of a data processing method provided by an exemplary embodiment, the method comprising:

[0083] Step 101: Acquire data to be processed, and determine a window column and a data column in the data to be processed, wherein the window column is used to indicate that the data to be processed is divided into windows according to the data in the column.

[0084] When using the data processing method provided by this application, the data to be processed can be a part of all the data, which is more flexible than sorting all the data at once. For example, in some scenarios, it is necessary to divide the sales data for the whole year into windows according to the specified window columns. In this case, the data to be processed can be the sales data of each quarter. For example, if the data to be processed is the sales data of the first and second quarters, when processing the sales data of the third quarter, the sales data of the third quarter can be divided into windows based on the first and second quarters. There is no need to sort all the sales data of the first, second and third quarters, which can improve the window division performance and shorten the division time.

[0085] Step 102: Determine the window number of each row according to the window column data of each row in the data to be processed.

[0086] The window number of the corresponding row is determined based on the window column data in each row of the window column. The window numbers obtained based on the window column data in each row of the same window are the same.

[0087] Step 103: compress and store the window column data according to the window number of each row.

[0088] A mapping relationship between the window number and the window column data is established. For the same window, the window column data corresponding to the same window number is stored only once, which can reduce memory usage.

[0089] Step 104: store the index of each row of data column data as a linked list according to the window number of each row, and store the linked list headers corresponding to different window numbers in the linked list header, wherein the index in the linked list is used to indicate the index jump information of the same window number.

[0090] The index of each row and column of data is stored as a linked list according to the window number. The index in the linked list points to the location of the next data column with the same window number. The same window number is recorded as a window, and different window numbers correspond to different windows. The linked list stores window chains corresponding to multiple windows, and the linked list header corresponding to each window chain is stored in the linked list header. If there are N different window numbers, there will be N window chains corresponding to N window chains. N window chains have N linked list headers. The jump information of each window chain in the N window chains is stored in the same linked list. Where N is a positive integer.

[0091] This application determines the window number of each row through the window column data of each row in the data to be processed, and compresses and stores the window column data according to the window number of each row, which can save memory. According to the window number of each row, the index jump information of the same window number is stored as a linked list, and the linked list headers corresponding to different window numbers are stored in the linked list header, avoiding the workload of full sorting and shortening the time occupied by window division.

[0092] The above is an introduction to the basic implementation process of this application. The following will introduce multiple optional implementation methods of this application.

[0093] In some embodiments, the present application may calculate a hash value for each row of window column data and insert the hash value into a hash table to obtain the window number of each row.

[0094] In a possible implementation, the process of determining the window number of each row according to the window column data of each row in the data to be processed may include the following steps:

[0095] Step 102 - 1 : Determine the hash value of each row and window column data in the data to be processed.

[0096] Step 102-2: Insert the hash value into the hash table, and determine the window number based on the position of the hash value in the hash table.

[0097] In a possible implementation, step 100 - 2 includes two cases in which the hash value is inserted into the hash table and the window number is determined according to the position of the hash value in the hash table. The two cases are described below respectively.

[0098] In the case that the hash value does not exist in the hash table, the window number is determined according to the position of the hash value in the hash table, and a mapping relationship between the window number and the hash value is established.

[0099] When a hash value already exists in the hash table, the window number corresponding to the hash value is determined according to the mapping relationship.

[0100] When using a hash table to determine the window number, a hash table query is used before the hash value corresponding to the window column data is inserted into the hash table. This allows for quick determination of whether the same hash value has already been inserted into the hash table without requiring a traversal query, thus shortening the query time. Therefore, this application improves the performance of window partitioning through the hash table and provides a significant memory compression opportunity: for the same window, its window column data is effectively the same, so for each window, the window column data can be stored only once, thereby compressing memory and avoiding redundancy.

[0101] For ease of understanding, this application Figure 2 The data to be processed shown in the figure is used as an example to illustrate the data processing method provided by this application. Figure 2 The data to be processed is 5 rows and 2 columns of data, one of which is a window column, that is, the data to be processed is divided into different windows according to the data in this column; and the other column is a data column.

[0102] According to the window column data a in the window column of the first row, determine that the window number of the first row is 1. Specifically, calculate the hash value of the window column data a, insert the hash value into the hash table, and determine the window number 1 corresponding to the window column data a according to the position of the hash value corresponding to the window column data a in the hash table. Window number 1 can be understood as the first hash value inserted into the hash table. Construct a mapping relationship between window column data a and window number 1. Similarly, according to the window column data b in the window column of the second row, determine that the window number of the second row is 2, and construct a mapping relationship between window column data b and window number 2. When determining the window number of the third row, since the hash value corresponding to the window column data a has been inserted into the hash table, the window number of the third row can be determined based on the mapping relationship between window column data a and window number 1. As shown in FIG. Figure 2 As shown in the figure, after the above processing, the obtained window column (after compression) is compressed from 5 rows to 2 rows, reducing memory usage. In this example, the window number is determined by traversing each row of data. In addition, the window column can also be processed in batches. Specifically, after determining the window number corresponding to one row in the same window, the window numbers of other rows in the same window are obtained. This application does not make specific limitations.

[0103] Currently, the DuckDB window operator's window partitioning method also partially adopts a hash table approach, but it still primarily uses a sorting algorithm, with the hash table only used to assist the sorting algorithm. In DuckDB, a hash table is first used to partition all data into fixed partitions (e.g., 1024 partitions). A sorting algorithm is then used to sort and partition these fixed partitions. Essentially, the partitioning is still performed using a sorting algorithm, with the hash table only used for preliminary data segmentation, avoiding full sorting and reducing the workload of the sorting algorithm. Therefore, DuckDB's performance improvement for window partitioning is relatively limited. Furthermore, the hash table introduced in DuckDB requires additional memory, resulting in increased memory consumption. This application utilizes memory compression when using a hash table for partitioning, meaning that the window column data for the same window only needs to be stored once. This unique memory compression opportunity offered by hash table partitioning can reduce the memory usage of this application. When the number of windows is small and the number of rows in a window is large, it can even use less memory than the original sorting-based partitioning algorithm. In other words, DuckDB uses a hash table only to assist with sorting, and essentially, the partitioning is still performed using a sorting algorithm. However, this application completely uses hash tables to perform window division, and does not require sorting operations during the division process, thereby reducing complexity and improving performance.

[0104] Need to explain, Figure 2The data to be processed shown in FIG includes one window column and one data column, but the data to be processed in this application may include multiple window columns and / or multiple data columns. If there are multiple window columns, the multiple window columns in a row are considered as a whole to determine the window number. If there are multiple data columns, the multiple data columns in a row are considered as a whole for subsequent processing.

[0105] It should be noted that the above process determines the window number through the hash value corresponding to the window column data. In other embodiments, a mapping relationship between the window column data and the window number can be directly established, and then the window number is determined based on the mapping relationship between the window column data and the window number.

[0106] In another possible implementation, the window number may be determined directly according to the mapping relationship between the window column data and the window number.

[0107] Taking the 5-row, 2-column data to be processed as an example, we can see from the window column of the data to be processed that it contains two different window column data. We then establish a mapping relationship between window column data a and window number 1, and a mapping relationship between window column data b and window number 2. When determining the window number for a row, we query the mapping relationship to determine the row's window number. This approach is suitable for data with relatively simple window column data. For complex window column data, using a hash function offers higher efficiency and performance.

[0108] The above embodiment illustrates the process of determining the window number. The following embodiment will illustrate the process of storing the index of each row of data column data as a linked list according to the window number of each row, and storing the linked list headers corresponding to different window numbers in the linked list header.

[0109] In some embodiments, step 104 stores the index of the data column of each row as a linked list according to the window number of each row, and stores the linked list header corresponding to different window numbers at the linked list header, including: performing the following operations for each row:

[0110] Step 104-1: Determine the linked list head corresponding to the window number according to the window number of the row.

[0111] Step 104-2: Store the index stored in the linked list header into the linked list corresponding to the row.

[0112] Step 104-3: Store the row index in the linked list header.

[0113] This application can further reduce memory overhead by creating a linked list, storing only the linked list header, and storing the row index in the linked list header, that is, inserting data in the linked list header.

[0114] Still taking the data to be processed with 5 rows and 2 columns as an example, the storage process of the above linked list and linked list header is explained.

[0115] like Figure 2 As shown, since the window numbers in the same window are the same, the linked list header corresponding to the window number is constructed according to the number of window numbers, as shown in Figure 3 As shown, there are window numbers 1 and 2, and two linked list headers are constructed. A flag is set in the constructed linked list header. In this example, the flag is -1, indicating that there is no row with the same window number as the row.

[0116] The linked list header is determined based on the window number. Row indexes with the same window number form a window chain. The linked list header here refers to the linked list head of the window chain, so in this implementation, the linked list header is also called the window linked list header. A linked list is constructed to store the index, which is also called the index linked list.

[0117] According to the window number 1 in the first row, the linked list head of the window chain W1 corresponding to window number 1 is determined to be -1. The index -1 stored in the linked list head is stored in the linked list corresponding to the first row, and the index 1 of the first row is stored in the linked list head of the window chain W1. After completing the processing of the first row, the following is obtained: Figure 4 The effect shown.

[0118] According to the window number 2 in the second row, the linked list head of the window chain W2 corresponding to window number 2 is determined to be -1. The index -1 stored in the linked list head is stored in the linked list corresponding to the second row, and the index 2 of the second row is stored in the linked list head of the window chain W2. After completing the processing of the second row, the following is obtained: Figure 5 The effect shown.

[0119] According to the window number 1 in the third row, the linked list head of the window chain W1 corresponding to window number 1 is determined to be 1. The index 1 stored in the linked list head is stored in the linked list corresponding to the third row. The index 3 of the third row is stored in the linked list head of the window chain W1. After completing the processing of the third row, the following is obtained: Figure 6 The effect shown. Figure 6 As can be seen in the figure shown, the third row in the linked list records the jump information from the third row to the first row.

[0120] According to the window number 2 in the fourth row, the linked list head of the window chain W2 corresponding to window number 2 is determined to be 2. The index 2 stored in the linked list head is stored in the linked list corresponding to the fourth row. The index 4 of the fourth row is stored in the linked list head of the window chain W2. After completing the processing of the fourth row, the following is obtained: Figure 7 The effect shown. Figure 7 As can be seen from the figure, the linked list not only records the jump information from the third row to the first row, but also records the jump information of the window chain W2 from the fourth row to the second row.

[0121] According to the window number 2 in the fifth row, the linked list head of the window chain W2 corresponding to window number 2 is determined to be 4. The index 4 stored in the linked list head is stored in the linked list corresponding to the fifth row, and the index 5 of the fifth row is stored in the linked list head of the window chain W2. After completing the processing of the fifth row, the following is obtained: Figure 8 The effect shown.

[0122] pass Figures 3 to 8 As shown in the diagram, the index linked list and window linked list header arrays are used to encode window information for each row. The indexes of the same window are organized into a linked list: the index linked list contains linked list jump information, while the window linked list header records the linked list headers for each window. Window information refers to the window to which the row belongs. For example, for the third row of data to be processed in the example above, the window information for the third row includes that the third row belongs to the first window and that the previous row, which also belongs to the first window, is the first row. When encoding a row of data, the current linked list header for that window is found based on the row's window number. The row of data is then inserted from the linked list header. If the linked list header is empty, the row becomes the header; otherwise, the existing linked list header is set to point to the new row, and the new row is set as the new header. This insertion method from the linked list header allows a single linked list header to represent a window chain, eliminating the need to store additional linked list tails and thus saving memory. In the example disclosed herein, two windows W1 and W2 are obtained by construction, that is, window information of each row is encoded into a window list through an index list and a window list header.

[0123] In one possible implementation, an index column is constructed to store the index of each row of data column data. The index is the row number corresponding to each row of data column data. After the index column is constructed, the index of each row of data column data is stored as a linked list based on the index stored in the index column.

[0124] Still taking the data to be processed with 5 rows and 2 columns as an example, Figure 9 As shown, after obtaining the window number for each row, an index column, index linked list, and window linked list header are constructed. The index column is actually the row number of the data to be processed, for example, the index of the first row is 1, and the index of the second row is 2. The index column is constructed to facilitate subsequent sorting and to assist in subsequent operations of the window operator. The process of constructing the window linked list headers for the index linked list has been described in the previous embodiment and will not be repeated here.

[0125] In some embodiments, Figure 9In the scenario shown, a sixth row of data is added: the window column data is c, and the data column data is F. Based on the window column data c in the sixth row, the window number in the sixth row is determined to be 3, and c is added to the window column (after compression). Since there is no linked list header corresponding to window number 3, a linked list header for window number 3 is constructed. The subsequent process is the same as that for rows 1 to 5 and is not repeated here.

[0126] The above process encodes the data to be processed, i.e., the data input process. During the data input phase, the data to be processed is input and divided, compressed, indexed, and stored. During the data input phase, the partitioning information for all windows has been calculated. The compressed and indexed data preserves the window information of the original data while compressing redundant data, thereby reducing memory usage.

[0127] The following describes the data output process. During the data output process, the data is decoded and restored, and the decoded data is output in each window in sequence.

[0128] In some embodiments, the data processing method of the present application further includes the following steps:

[0129] Step 201: Build a window index column based on the index in the linked list header and the index jump information of the linked list indicating the same window number.

[0130] For each window, since the linked list for that window has already been constructed, we can construct an index column for that window based on it. The process of constructing the window index column involves starting from the window's linked list header and accessing each linked list item one by one based on the index linked list. The index numbers are then added to the window index column. Note that each linked list header corresponds to one window index column.

[0131] Step 202: Get the data column data according to the index in the window index column.

[0132] Step 203: Obtain data in the same window according to the data column data and the window column data corresponding to the window number.

[0133] In some embodiments, step 201 constructs a window index column based on the index in the linked list header and the index jump information of the linked list indicating the same window number, including the following steps:

[0134] Step 201-1: Obtain the first index in the linked list header, where the first index corresponds to the first window number.

[0135] Step 201-2: Obtain a second index in the linked list according to the row indicated by the first index.

[0136] Step 201-3: perform index jump in the linked list based on the second index, and obtain a third index in the linked list indicating the first window number.

[0137] Step 201-4: Obtain a window index column corresponding to the first window number according to the first index, the second index, and the third index.

[0138] Still taking the data to be processed with 5 rows and 2 columns as an example, Figure 9 As shown, get the first index 3 in the linked list header, which corresponds to the first window. According to the third row indicated by the first index 3, get the second index 1 corresponding to the third row in the linked list. Then, according to the first row indicated by the second index 1, get the third index -1 corresponding to the first row in the linked list. The third index is a flag bit -1, indicating that the first row is the last row in the first window. Execute the above process to get Figure 10a The W1 window index column shown in . Get the next first index 5 in the linked list header in the same way, jump to the index in the linked list according to the instruction of the first index 5, get all the indexes indicating the second window number in the linked list, and get Figure 10b The W2 window index column is shown in .

[0139] For each window, after obtaining the window index column, the window data can be further read according to the index. For the data column, the data column in the data to be processed can be read according to the index in the window index column. For the window column, the compressed window column is read according to the window number. Figure 10a and Figure 10b As shown in FIG, the process of constructing a window index column is called constructing a partition index item, and the partition data is read after the partition index item is constructed.

[0140] In the use case of window operators, after constructing the window index column, the window data is often not read directly. Instead, further operations are performed on the window data, such as sorting within the window, calculating window functions, etc. The window index column constructed in this application allows access to data before it is materialized, conveniently providing a flexible way to read data while avoiding data copying.

[0141] Compared with traditional sorting algorithms, first, this application uses a hash table algorithm for window division, which greatly improves the performance of window division; second, when using the hash table algorithm for window division, this application only stores the window column data in the same window once, that is, memory compression is performed, reducing memory occupancy. When the number of windows is small, the memory occupancy is even lower than that of the sorting algorithm, thus having advantages in both performance and resource consumption.

[0142] This application uses a hash table for window division, which can shorten the time taken for window division. In addition, by compressing and storing window column data, the memory occupied by the window column is reduced to improve data processing performance, thereby significantly reducing the response time to the user and improving the user's query experience.

[0143] Corresponding to the aforementioned method embodiments, this specification also provides embodiments of an apparatus and a computing device to which the apparatus is applied.

[0144] See also Figure 11 , Figure 11 1 is a block diagram of a data processing device provided by an exemplary embodiment, the device comprising:

[0145] An acquiring unit 1101 is configured to acquire data to be processed and determine a window column and a data column in the data to be processed, wherein the window column is used to indicate that the data to be processed is divided into windows according to the column data.

[0146] A determining unit 1102 is configured to determine a window number for each row according to window column data for each row in the data to be processed;

[0147] A compression storage unit 1103 is used to compress and store window column data according to the window number of each row;

[0148] The first processing unit 1104 is configured to store the index of the data column of each row as a linked list according to the window number of each row, and store the linked list headers corresponding to different window numbers in the linked list header, wherein the index in the linked list is used to indicate index jump information of the same window number.

[0149] In one embodiment of this specification, the determining unit 1102 is configured to:

[0150] Determine a hash value of each row of window column data in the data to be processed;

[0151] The hash value is inserted into a hash table, and the window number is determined based on the position of the hash value in the hash table.

[0152] In one embodiment of the present specification, the determining unit 1102, when inserting the hash value into the hash table and determining the window number according to the position of the hash value in the hash table, is configured to:

[0153] If the hash value does not exist in the hash table, determining the window number according to the position of the hash value in the hash table, and establishing a mapping relationship between the window number and the hash value;

[0154] In the case that a hash value already exists in the hash table, the window number corresponding to the hash value is determined according to the mapping relationship.

[0155] In one embodiment of the present specification, the first processing unit 1104 is configured to perform the following operations for each row:

[0156] According to the window number of the row, determine the linked list head corresponding to the window number;

[0157] Storing the index stored in the linked list header into the linked list corresponding to the row;

[0158] The index of the row is stored in the head of the linked list.

[0159] In one embodiment of the present specification, before storing the index of each row and column of data as a linked list, the apparatus further includes: a construction unit 1105 for constructing an index column, in which the index of each row and column of data is stored.

[0160] In one embodiment of the present specification, the apparatus further includes a second processing unit 1106, configured to:

[0161] Constructing a window index column according to the index in the linked list header and the index jump information of the linked list indicating the same window number;

[0162] Obtain data column data according to the index in the window index column;

[0163] The data in the same window is obtained according to the window column data corresponding to the data column data and the window number.

[0164] In one embodiment of the present specification, the second processing unit 1106, when constructing the window index column according to the index in the linked list header and the index jump information indicating the same window number in the linked list, is configured to:

[0165] Obtaining a first index in the linked list header, where the first index corresponds to a first window number;

[0166] According to the row indicated by the first index, obtain the second index in the linked list;

[0167] Performing an index jump in the linked list based on the second index to obtain a third index in the linked list indicating the first window number;

[0168] A window index column corresponding to the first window number is obtained according to the first index, the second index, and the third index.

[0169] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the units can be selected according to actual needs to achieve the purpose of the scheme of this specification. Those of ordinary skill in the art can understand and implement it without paying any creative work.

[0170] This application also provides a computing device, see Figure 12 , Figure 12 This is a schematic diagram of a computing device provided by an exemplary embodiment. Figure 12 At the hardware level, the device includes a processor 1202, an internal bus 1204, a network interface 1206, a memory 1208, and a non-volatile memory 1210. Of course, it may also include hardware required to implement other functions. One or more embodiments of this specification can be implemented based on software, such as the processor 1202 reading the corresponding computer program from the non-volatile memory 1210 into the memory 1208 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0171] The present application also provides a computer program product, including a computer program, which implements the data processing method provided in any embodiment of the present application when the computer program is executed by a processor.

[0172] The devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0173] In a typical configuration, a computer includes one or more processors (Central Processing Unit, CPU), input / output interfaces, network interfaces and memory.

[0174] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0175] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0176] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0177] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0178] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "an," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0179] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when..." or "when..." or "in response to determining."

[0180] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A data processing method, characterized in that: include: Acquire data to be processed, and determine a window column and a data column in the data to be processed, wherein the window column is used to indicate that the data to be processed is divided into windows according to the data in the column; Determining the window number of each row according to the window column data of each row in the data to be processed; Compress and store window column data according to the window number of each row; According to the window number of each row, the index of the data column of each row is stored as a linked list, and the linked list headers corresponding to different window numbers are stored in the linked list header, wherein the index in the linked list is used to indicate the index jump information of the same window number; Determining the window number of each row according to the window column data of each row in the data to be processed includes: Determine a hash value of each row of window column data in the data to be processed; Inserting the hash value into a hash table and determining the window number based on the position of the hash value in the hash table; Before storing the index of each row and column data as a linked list, it also includes: Constructing an index column, wherein the index of each row of data column data is stored in the index column; The method further comprises: Constructing a window index column according to the index in the linked list header and the index jump information of the linked list indicating the same window number; Obtain data column data according to the index in the window index column; Obtaining data within the same window according to the data column data and the window column data corresponding to the window number; Performs window sorting and window function calculations on data within the same window based on the index column.

2. The method according to claim 1, characterized in that Inserting the hash value into the hash table and determining the window number according to the position of the hash value in the hash table includes: If the hash value does not exist in the hash table, determining the window number according to the position of the hash value in the hash table, and establishing a mapping relationship between the window number and the hash value; In the case that a hash value already exists in the hash table, the window number corresponding to the hash value is determined according to the mapping relationship.

3. The method according to claim 1, characterized in that According to the window number of each row, the index of the data column of each row is stored as a linked list, and the linked list headers corresponding to different window numbers are stored in the linked list header, including: For each row, do the following: According to the window number of the row, determine the linked list head corresponding to the window number; Storing the index stored in the linked list header into the linked list corresponding to the row; The index of the row is stored in the head of the linked list.

4. The method according to claim 1, wherein The step of constructing a window index column according to the index in the linked list header and the index jump information of the linked list indicating the same window number includes: Obtaining a first index in the linked list header, where the first index corresponds to a first window number; According to the row indicated by the first index, obtain the second index in the linked list; Performing an index jump in the linked list based on the second index to obtain a third index in the linked list indicating the first window number; A window index column corresponding to the first window number is obtained according to the first index, the second index, and the third index.

5. A data processing device, characterized in that: The device comprises: an acquiring unit, configured to acquire data to be processed and determine a window column and a data column in the data to be processed, wherein the window column is used to indicate that the data to be processed is divided into windows according to the data in the column; a determining unit, configured to determine a window number of each row according to window column data of each row in the data to be processed; A compression storage unit, used for compressing and storing window column data according to the window number of each row; A first processing unit is configured to store the index of the data column of each row into a linked list according to the window number of each row, and store the linked list headers corresponding to different window numbers in the linked list header, wherein the index in the linked list is used to indicate index jump information of the same window number; Before storing the index of each row and column data as a linked list, it also includes: Build an index column that stores the index of each row of data column data The device further comprises: Constructing a window index column according to the index in the linked list header and the index jump information of the linked list indicating the same window number; Obtain data column data according to the index in the window index column; Obtaining data within the same window according to the data column data and the window column data corresponding to the window number; Performs window sorting and window function calculations on data within the same window based on the index column.

6. A computing device, characterized in that include: processor; a memory for storing processor-executable instructions; The processor implements the data processing method according to any one of claims 1 to 4 by running the executable instructions.

7. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instructions are executed by a processor, the data processing method according to any one of claims 1 to 4 is implemented.

Images