Data processing method, device, equipment and storage medium

By using the batch processing characteristics of columnar databases and SIMD instruction sets for parallel processing in data query, the problems of low data processing efficiency and slow query speed caused by the iterator model are solved, and more efficient data query is achieved.

CN112948413BActive Publication Date: 2024-12-03ALIBABA CLOUD COMPUTING CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201911259423.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-12-10
Publication Date
2024-12-03
Estimated Expiration
2039-12-10

AI Technical Summary

Technical Problem

The iterator model causes multi-layer function calls and excessive write and output operations during data query, resulting in low data processing efficiency and slow SQL statement query speed.

Method used

By obtaining the target model corresponding to the expression in the data query request, using the characteristics of the columnar database to divide the expression and target data into batches, reducing the number of writes and outputs, and using the SIMD instruction set for parallel processing, optimizing expression formula operation.

Benefits of technology

It improves data processing efficiency, reduces the amount of calculation, and significantly improves the data query speed of SQL statements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112948413B_ABST
    Figure CN112948413B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention provides a data processing method, device, equipment and storage medium, wherein the method may include: first, obtaining a target model corresponding to an expression in a data query request, wherein the expression includes a constraint condition; second, inputting the expression and target data in a columnar database into the target model to obtain an execution result corresponding to the constraint condition; wherein the target data includes all data in at least one column and part of data in at least one column; and then, outputting the execution result. This is used to solve the problem in the related art that the data processing efficiency is too low, which affects the slow data query speed of SQL statements.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a data processing method, device, equipment and storage medium. Background Art

[0002] When users use Structured Query Language (SQL) statements to query data, they can use the iterator model to query data. In the iterator model, the expression takes an abstract iterator as input, and obtains the data of each data table in the database row by row through the iterator, thereby calculating the result of each row, and outputting multiple rows of results as output to obtain the corresponding data.

[0003] However, this abstract iterator will cause performance loss, because during the iteration process of the iterator, the acquisition of each row of data will trigger multiple layers of function calls. At the same time, acquiring data row by row will cause excessive writing and output, consume more resources, and cause excessive calculation, thereby reducing data processing efficiency and reducing the data query speed of SQL statements. Summary of the invention

[0004] One or more embodiments of the present invention describe a data processing method, apparatus, device and storage medium to solve the problem in the related art that data processing efficiency is too low, which affects the slow data query speed of SQL statements.

[0005] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:

[0006] According to a first aspect, a data processing method is provided, which may include:

[0007] Get the target model corresponding to the expression in the data query request, where the expression contains constraints;

[0008] Inputting the expression and the target data in the column-based database into the target model to obtain an execution result corresponding to the constraint condition; wherein the target data includes all the data in at least one column and part of the data in at least one column;

[0009] Output the execution results.

[0010] According to a second aspect, a data processing device is provided, which may include:

[0011] An acquisition module is used to acquire a target model corresponding to an expression in a data query request, where the expression includes a constraint condition;

[0012] A processing module, used for inputting the expression and the target data in the column-based database into the target model to obtain an execution result corresponding to the constraint condition; wherein the target data includes all the data in at least one column and part of the data in at least one column;

[0013] Output module, used to output execution results.

[0014] According to a third aspect, a computing device is provided, the device comprising at least one processor and a memory, the memory being used to store computer program instructions, and the processor being used to execute the program in the memory to control the computing device to implement the data processing method as shown in the first aspect.

[0015] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. If the computer program is executed in a computer, the computer is caused to execute the data processing method shown in the first aspect.

[0016] In the embodiment of the present invention, a target model corresponding to an expression in a data query request is obtained, wherein the expression includes a constraint condition; the expression and the target data in a column-based database are input into the target model to obtain an execution result corresponding to the constraint condition. Here, the embodiment of the present invention makes full use of the characteristic that a column-based database can quickly obtain a column of data, reduces the number of writes or outputs in expression calculations, greatly reduces the time for obtaining data, reduces the amount of calculation, improves data processing efficiency, and improves the data query speed of SQL statements. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present invention may be better understood from the following description of specific embodiments of the present invention in conjunction with the accompanying drawings, wherein the same or similar reference numerals represent the same or similar features.

[0018] Figure 1 A schematic diagram showing an architecture of a data processing method according to an embodiment;

[0019] Figure 2 A flow chart showing a data processing method according to an embodiment;

[0020] Figure 3 A flowchart showing a first target model implementing a data processing method according to an embodiment;

[0021] Figure 4 A flowchart showing a method for implementing data processing using a second target model according to an embodiment;

[0022] Figure 5 A flowchart showing a method for implementing data processing using a first target model and a second target model according to an embodiment;

[0023] Figure 6 A flowchart showing a data processing method implemented by a third target model according to an embodiment;

[0024] Figure 7 A flowchart showing a fourth target model implementing data processing method according to an embodiment;

[0025] Figure 8 A flowchart showing a fifth target model implementing data processing method according to an embodiment;

[0026] Fig. 9 A flowchart showing another fifth target model implementing data processing method according to one embodiment;

[0027] Fig.10 A structural block diagram of a data processing device according to an embodiment is shown;

[0028] Fig.11 A schematic diagram of the structure of a computing device according to an embodiment is shown. DETAILED DESCRIPTION

[0029] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the specific embodiments described herein are only configured to explain the present invention and are not configured to limit the present invention. For those skilled in the art, the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present invention by illustrating examples of the present invention.

[0030] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such measured relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0031] Currently, in the iterator model, an expression takes an abstract iterator as input and outputs the calculation result. When using iterators to calculate expressions, the data in the data table in the row-based database is processed row by row. For each row in the table, an iterator can be used to obtain the data in this row, perform calculations, and then write back the results. After processing a row of data, the iterator will be moved to the next row and perform the same calculation. This operation often forms a loop until the iterator scans all rows, at which point the loop terminates, the expression calculation jumps to the next model, and then a similar loop is performed on this model. Because the iterator abstracts the entire data table, the abstract iterator will perform multiple layers of function calls when obtaining data, reducing the overall processing performance. In addition, each movement of the iterator involves a data fetch operation, which often involves too much writing and output, resulting in very large overhead and excessive calculation, thereby reducing data processing efficiency and the data query speed of SQL statements.

[0032] In order to solve the above technical problems, an embodiment of the present invention provides a data processing method, device, equipment and storage medium, as shown below.

[0033] First, a data processing architecture provided by an embodiment of the present invention is described.

[0034] like Figure 1 As shown, the architecture may include a target node and a column-based database. When the target node receives a data query request, it determines the target model corresponding to the expression in the data query request, where the expression includes a constraint condition; then, the expression and the target data in the column-based database are input into the target model to obtain an execution result corresponding to the constraint condition; wherein the target data includes all data in at least one column and part of data in at least one column; and outputs the execution result.

[0035] Among them, the column-based database in the embodiment of the present invention may include the Hisore column-based database. The Histore column-based database is a column-based database evolved from an open source database. The data of each data table in the database is stored in columns, so that the data of all rows of a certain column can be quickly obtained when the expression is calculated. This makes it naturally have the characteristic of batch processing of data. Therefore, the new batch-based target model in the embodiment of the present invention uses this feature to divide the data in the data table into different batches, and each expression processes the data in batches, thereby speeding up the processing speed of the entire expression.

[0036] In addition, the above architecture can be applied to application scenarios where users use SQL statements to query data, or can be applied to scenarios where expression calculations are performed based on the above target model.

[0037] Thus, by obtaining the target model corresponding to the expression in the data query request, the expression and the target data in the column-based database are input into the target model, and the execution result corresponding to the constraint condition of the expression is obtained. Here, the embodiment of the present invention makes full use of the characteristic that the column-based database can quickly obtain a column of data, reduces the number of writes or outputs in the expression operation, greatly reduces the time for obtaining data, reduces the amount of calculation, and improves data processing efficiency.

[0038] Here, based on the above architecture and application scenarios, the embodiments of the present invention are combined with Figure 2-Figure 9 The data processing method provided by the embodiment of the present invention is further explained.

[0039] Figure 2 A flow chart of a data processing method according to an embodiment is shown.

[0040] like Figure 2 As shown, the method may include steps 210 to 230: first, in step 210, a target model corresponding to an expression in a data query request is obtained, where the expression includes constraints; second, in step 220, the expression and target data in a columnar database are input into the target model to obtain an execution result corresponding to the constraints; then, in step 230, the execution result is output.

[0041] The following are detailed descriptions of the above steps:

[0042] First, involving step 210, the target node receives a data query request;

[0043] The data query request includes an expression, which can be expressed as "count the number of clicks on a certain product", "query the number of visits to a certain platform", etc. The expression may include at least one constraint condition, so the above expression "count the number of clicks on a certain product" may include the following constraints: "count the number of clicks on a certain product from January to June", "count the number of clicks on a certain product within 1-1000", etc.

[0044] After receiving the data query request, the method provided by the embodiment of the present invention may further include: determining, according to the data query request, a target model corresponding to the expression in the data query request.

[0045] Here, each expression has its corresponding target model. Six target models are provided in the embodiment of the present invention, which are described in detail below in conjunction with step 220.

[0046] Next, in step 220, the target data in the embodiment of the present invention includes all data in at least one column and part of data in at least one column. In addition, the data in the column-based database is stored in columns, and each column stores data of at least one attribute separately.

[0047] Based on this, in a possible embodiment, this step may specifically include: inputting the expression and the target data into the target model;

[0048] Based on the target model, the Single Instruction Multiple Data (SIMD) instruction set is used to parallelize the expression and target data to obtain the execution results corresponding to the constraints.

[0049] Therefore, by using batch calculation, the SIMD instruction set can be fully utilized to perform expression calculations, thereby speeding up the calculation speed.

[0050] In another possible embodiment, this step may specifically include inputting the first bitmap data, the expression and the target data into the target model;

[0051] Filter the target data using the first bitmap data to obtain data that meets the constraint conditions;

[0052] Based on the target model and the first bitmap data, the data satisfying the constraint conditions are processed to obtain the execution results corresponding to the constraint conditions.

[0053] Therefore, by using the input filtering first bitmap data (mask) during the processing, that is, by preferentially filtering out some data that do not meet the constraint conditions, the number of expression operations can be reduced.

[0054] It should be noted that the second possible embodiment described above can be implemented based on the first possible embodiment, that is, the first bitmap data, the expression and the target data are input into the target model;

[0055] Filter the target data using the first bitmap data to obtain data that meets the constraint conditions;

[0056] Based on the target model and the first bitmap data, the Single Instruction Multiple Data (SIMD) instruction set is used to parallelize the expression and the target data to obtain the execution result corresponding to the constraint condition.

[0057] In this way, the target data can be processed in batches, making full use of the columnar database's ability to quickly obtain a column of data, reducing the number of writes and outputs in expression operations, greatly reducing the time to query data results, and being cache-friendly. These optimizations can improve the overall performance of the entire expression. In addition, by using the incoming filter mask during the processing process to preferentially filter out some data that does not meet the constraints, the number of expression operations can be greatly reduced. At the same time, by batch calculation, the SIMD instruction set can be fully utilized for expression operations, speeding up the operation speed.

[0058] Based on the above content, an embodiment of the present invention combines at least one of the following target models to explain in detail the above step of "processing the data satisfying the constraint conditions based on the target model and the first bitmap data to obtain an execution result corresponding to the constraint conditions".

[0059] (1) When the target model is the first target model, the target data includes a first column of data and a second column of data, and the first column of data and the second column of data have the same number of rows, the step may specifically include:

[0060] Based on the first target model, the first column data and the second column data having the same row are added to obtain the third column data; wherein the third column data and the first column data have the same number of rows;

[0061] The first bitmap data and the third column data are determined as the execution result corresponding to the constraint condition.

[0062] The input of the first target model may include the first bitmap data, the expression, the first column data and the second column data. Through the above data processing method, the output of the first target model may include the first bitmap data and the third column data.

[0063] For example, when the first target model is a summation model, Figure 3As shown, a tree structure with an expression of a+b and its implementation model are shown. Where a and b are any two columns in the target data, namely the first column data and the second column data. Based on this, the expression "+", mask, a and b are used as input, the data of a and b in each row are added, and then the result is output. In the first target model based on batch processing, the expression in the first target model receives a mask for filtering, a batch of data in column a, and a batch of data in column b. The role of mask is to filter out the rows that have been filtered (that is, it can be understood as rows that do not meet the constraints; and / or rows that meet the preset conditions that have been marked), and then add the remaining rows in batches, and finally output a mask and the result a+b of this batch of data. In this expression, since the input mask is not changed (that is, the mask is not marked to hide 0, and the mask only includes visible rows, namely marked 1), the input mask can be directly used as output.

[0064] (2) When the target model is the second target model, the target data includes the fourth column data and the constant data, and the fourth column data and the constant data have the same number of rows, the step may specifically include:

[0065] When the constraint condition is to select data greater than the constant data in the fourth column data, based on the second target model, the fourth column data with the same number of rows and the constant data are compared to obtain an execution result corresponding to the constraint condition;

[0066] The execution result includes the second bitmap data, which includes the visible row corresponding to the row where the fourth column data with the same row number as the first bitmap data is greater than the constant data, and the second bitmap data and the fourth column data have the same number of rows.

[0067] The input of the second target model may include the first bitmap data, the expression, the fourth column data and the constant data. Through the above data processing method, the output of the second target model may include the second bitmap data.

[0068] For example, Figure 4As shown, the tree structure of the expression fourth column data a> constant data 3 and its implementation model are shown. Based on this, in the second target model based on batch processing, the expression in the second target model receives a mask, a batch of data in column a, and a batch of data with an amount of data equal to the constant 3 of a as input. The mask outputted in the result is based on the input mask. After using the mask to filter out some rows (i.e., rows that do not meet the constraints), compare a>3 in batches. If a row is less than or equal to 3, reset these rows in the result mask (i.e., mark the rows that do not meet the constraints as hidden rows 0, and mark the rows that meet the constraints and meet the comparison results as visible rows 1), and finally output a result mask (including visible rows 1 that meet the constraints and meet the comparison results).

[0069] (3) When the target model is a first target model and a second target model, the target data includes a first column of data and a second column of data, and the target model includes a first model and a second model; the target data includes first target data and second target data, the first target data includes a fifth column of data and a sixth column of data, and the fifth column of data and the sixth column of data have the same number of rows, this step may specifically include:

[0070] Based on the first model, the fifth column data and the sixth column data having the same row are added to obtain the seventh column data; wherein the seventh column data and the fifth column data have the same number of rows;

[0071] Inputting the first bitmap data, the seventh column data and the second target data into the second model; wherein the second target data includes the eighth column data, and the eighth column data and the seventh column data have the same number of rows;

[0072] When the constraint condition is to select data in the seventh column that is greater than the data in the eighth column, based on the second model, the seventh column data and the eighth column data having the same number of rows are compared to obtain an execution result corresponding to the constraint condition;

[0073] The execution result includes the third bitmap data, which includes the visible row corresponding to the row where the seventh column data with the same row number as the first bitmap data is greater than the eighth column data, and the third bitmap data has the same number of rows as the seventh column data.

[0074] For example, Figure 5 As shown in the figure, a combined expression (a+b)>second target data c is shown. This expression is composed of (1) and (2). The mask and (a+b) result output in (1) can be used as inputs of the expression (a+b)>c. In addition, a batch of data with the same amount of data is taken from c as input, and finally the (a+b)>c operation is performed in batches to obtain a result mask.

[0075] Thus, based on the first target model and the second target model, by using the incoming filtered first bitmap data (mask) during the processing and inputting the resulting mask into the next node (which may refer to the second target model and / or other target nodes), the number of expression operations can be greatly reduced.

[0076] (4) When the target model is the third target model, the target data includes the ninth column of data, the first bitmap data includes hidden rows and visible rows, and the first bitmap data and the ninth column of data have the same number of rows, this step may specifically include:

[0077] Based on the target model, the hidden rows in the ninth column of data are filtered using the hidden rows to obtain the data in the ninth column of data that have the same rows as the visible rows;

[0078] When the expression is data summation, the data of the same row in the visible rows are accumulated to obtain the accumulated data;

[0079] The accumulated data and the first bitmap data are determined as execution results corresponding to the constraint conditions.

[0080] The input of the third target model may include the first bitmap data, the expression and the ninth column data. Through the above data processing method, the output of the third target model may include the accumulated data and the first bitmap data.

[0081] For example, Figure 6 The figure shows the tree structure and implementation model of an aggregation query expression SUM. This expression takes a mask for filtering and the ninth column data a as input. After filtering out the row data corresponding to row 0 in the ninth column data using the rows marked as 0 in the mask, the remaining data (i.e. the rows corresponding to visible row 1) is summed, and finally a mask and the sum of this batch of data (sum(a)) are output.

[0082] (5) When the target model is the fourth target model, the target data includes the tenth column of data, the first bitmap data includes hidden rows and visible rows, and the first bitmap data and the tenth column of data have the same number of rows, this step may specifically include:

[0083] Based on the target model, the hidden rows in the tenth column of data are filtered using the hidden rows to obtain the data in the tenth column of data that have the same rows as the visible rows;

[0084] According to the first preset condition in the target model, the data in the tenth column of data that has the same row as the visible row is marked to obtain the marked eleventh column of data;

[0085] The first bitmap data and the eleventh column data are determined as the execution results corresponding to the constraint conditions.

[0086] Further, when the target model includes the first preset condition and the second preset condition; the visible row includes the first visible row and the second visible row;

[0087] According to the first preset condition in the target model, the data in the tenth column of data that has the same row as the visible row is marked to obtain the marked eleventh column of data, including:

[0088] According to the first preset condition, the data in the tenth column of data that has the same row as the first visible row is marked, and the first visible row is adjusted to a hidden row;

[0089] According to the second preset condition, the data in the tenth column of data that has the same row as the second visible row is marked until the third preset condition is met, and the marking is ended to obtain the marked eleventh column of data.

[0090] The input of the fourth target model may include the first bitmap data (in this example, the first bitmap data includes hidden row 0 and visible row 1), an expression (carrying 3 constraints) and the tenth column of data. Through the above data processing method, the output of the fourth target model may include the first bitmap data and the eleventh column of data.

[0091] For example, Figure 7 As shown in the figure, the tree structure of the conditional function (CASE WHEN) expression and its implementation model are shown. This expression contains three jump branches, namely conditional judgments. In the initial case, a mask is used to filter out part of the data (the rows shown by the black squares are hidden rows 0). In the result of the first conditional judgment (when a), the rows where the gray squares are located meet the conditions, so those rows are set to conditional result 5, and these rows are filtered out (that is, they are marked as hidden rows 0), and are not considered in the next conditional judgment. In the second conditional judgment (when b), the rows where the black squares are located are first filtered out, and then the calculation finds that the rows where the gray squares are located meet the second conditional judgment, so those rows are set to conditional result 6, and these rows are filtered out (that is, they are marked as hidden rows 0), and are not considered in the next conditional filtering. In the last conditional judgment, all the remaining rows (rows where the white squares are located) are considered by default, and these rows are set to conditional result 7. Finally, this expression outputs a mask and all conditional results that meet the constraints.

[0092] (6) When the target model is the fifth target model, this step may specifically include:

[0093] Determine, according to the first bitmap data, the expression and the target data, the fourth bitmap data corresponding to the first expression and the fifth bitmap data corresponding to the second expression; wherein the first expression or the second expression is related to any column of data in the target data;

[0094] Based on the target model, performing a logical operation on the fourth bitmap data and the fifth bitmap data to obtain sixth bitmap data;

[0095] The sixth bitmap data is determined as an execution result corresponding to the constraint condition.

[0096] The logic operation includes at least one of the following: an AND gate, an OR gate, and a NOT gate.

[0097] Here, it should be noted that the input in the fifth target model may include fourth bitmap data corresponding to the first expression and fifth bitmap data corresponding to the second expression, wherein the method for determining the fourth bitmap data and the fifth bitmap data can be based on the output of a result mask for input a in the above (2), which can be understood as the fourth bitmap data mask corresponding to the first expression a; similarly, if b is input through the above (2), then the output result mask for input b can be understood as the fifth bitmap data mask corresponding to the second expression a.

[0098] Based on this, when the logic operation is an AND gate, such as Figure 8 As shown, the tree structure of the expression containing AND and its implementation model are shown. The expression in the fifth target model receives the result mask of two sub-expressions as input (the mask output of expression a and the mask output of expression b), performs an AND operation on the masks of the two inputs, obtains the result mask and finally outputs it.

[0099] When the logic operation is an OR gate, such as Fig. 9 As shown, the tree structure of the expression containing OR and its implementation model are shown. The expression in the fifth target model receives the result mask of two sub-expressions as input (the mask output of expression a and the mask output of expression b), performs an OR operation on the two input masks, and obtains the result mask as the final output.

[0100] Thus, based on the second target model and the fifth target model, by using the incoming filtered first bitmap data (mask) during processing and inputting the resulting mask to the next node, the number of expression operations can be greatly reduced.

[0101] In summary, in the embodiment of the present invention, by obtaining a target model corresponding to an expression in a data query request, wherein the expression includes a constraint condition; the expression and the target data in the column-based database are input into the target model, and an execution result corresponding to the constraint condition is obtained. Here, the embodiment of the present invention makes full use of the characteristic that a column-based database can quickly obtain a column of data, reduces the number of writes or outputs in expression calculations, greatly reduces the time for obtaining data, reduces the amount of calculation, improves data processing efficiency, and improves the data query speed of SQL statements.

[0102] In addition, by using the incoming filter mask during the processing, that is, by filtering out some data that does not meet the constraints first, the number of expression operations can be greatly reduced. At the same time, by batch calculation, the SIMD instruction set can be fully utilized for expression calculation, which speeds up the calculation speed.

[0103] Based on the above data processing method, an embodiment of the present invention provides a data processing device. Fig.10 A structural block diagram of a data processing device according to an embodiment is shown.

[0104] like Fig.10 As shown, the device 1000 may specifically include:

[0105] The acquisition module 1001 is used to acquire a target model corresponding to an expression in a data query request, where the expression includes a constraint condition;

[0106] Processing module 1002, for inputting the expression and the target data in the column-based database into the target model to obtain an execution result corresponding to the constraint condition; wherein the target data includes all the data in at least one column and part of the data in at least one column;

[0107] The output module 1003 is used to output the execution result.

[0108] In a possible embodiment, the processing module 1002 may be specifically used to input the first bitmap data, the expression and the target data into the target model;

[0109] Filter the target data using the first bitmap data to obtain data that meets the constraint conditions;

[0110] Based on the target model and the first bitmap data, the data satisfying the constraint conditions are processed to obtain the execution results corresponding to the constraint conditions.

[0111] Furthermore, the embodiment of the present invention provides 6 possible target models for detailed description.

[0112] (1) When the target data includes a first column of data and a second column of data, and the first column of data and the second column of data have the same number of rows, the processing module 1002 can be specifically used to, based on the target model, add the first column of data and the second column of data having the same rows to obtain a third column of data; wherein the third column of data has the same number of rows as the first column of data; and determine the first bitmap data and the third column of data as the execution results corresponding to the constraint condition.

[0113] (2) When the target data includes the fourth column data and the constant data, and the fourth column data and the constant data have the same number of rows, the processing module 1002 can be specifically used to, when the constraint condition is to select data in the fourth column data that is greater than the constant data, compare the fourth column data and the constant data that have the same number of rows based on the target model to obtain an execution result corresponding to the constraint condition;

[0114] The execution result includes the second bitmap data, which includes the visible row corresponding to the row where the fourth column data with the same row number as the first bitmap data is greater than the constant data, and the second bitmap data and the fourth column data have the same number of rows.

[0115] (3) When the target model includes the first model and the second model; the target data includes the first target data and the second target data, the first target data includes the fifth column data and the sixth column data, and the fifth column data and the sixth column data have the same number of rows, the processing module 1002 can be specifically configured to, based on the first model, add the fifth column data and the sixth column data having the same row to obtain the seventh column data; wherein the seventh column data and the fifth column data have the same number of rows;

[0116] Inputting the first bitmap data, the seventh column data and the second target data into the second model; wherein the second target data includes the eighth column data, and the eighth column data and the seventh column data have the same number of rows;

[0117] When the constraint condition is to select data in the seventh column that is greater than the data in the eighth column, based on the second model, the seventh column data and the eighth column data having the same number of rows are compared to obtain an execution result corresponding to the constraint condition;

[0118] The execution result includes the third bitmap data, which includes the visible row corresponding to the row where the seventh column data with the same row number as the first bitmap data is greater than the eighth column data, and the third bitmap data has the same number of rows as the seventh column data.

[0119] (4) When the target data includes the ninth column of data, the first bitmap data includes hidden rows and visible rows, and the first bitmap data and the ninth column of data have the same number of rows, the processing module 1002 can be specifically used to, based on the target model, use the hidden rows to filter the hidden rows in the ninth column of data to obtain the data in the ninth column of data that have the same rows as the visible rows; when the expression is data summation, the data in the visible rows that have the same rows are added to obtain the accumulated data; and the accumulated data and the first bitmap data are determined as the execution results corresponding to the constraint conditions.

[0120] (5) When the target data includes the tenth column of data, the first bitmap data includes hidden rows and visible rows, and the first bitmap data and the tenth column of data have the same number of rows, the processing module 1002 can be specifically used to, based on the target model, use the hidden rows to filter the hidden rows in the tenth column of data to obtain the data in the tenth column of data that have the same rows as the visible rows; according to the first preset condition in the target model, mark the data in the tenth column of data that have the same rows as the visible rows to obtain the marked eleventh column of data; and determine the first bitmap data and the eleventh column of data as the execution results corresponding to the constraint condition.

[0121] Further, when the target model includes the first preset condition and the second preset condition; when the visible row includes the first visible row and the second visible row, the processing module 1002 can be specifically used to, according to the first preset condition, mark the data in the tenth column of data that has the same row as the first visible row, and adjust the first visible row to a hidden row;

[0122] According to the second preset condition, the data in the tenth column of data that has the same row as the second visible row is marked until the third preset condition is met, and the marking is ended to obtain the marked eleventh column of data.

[0123] (6) The processing module 1002 can be specifically used to determine, based on the first bitmap data, the expression and the target data, the fourth bitmap data corresponding to the first expression and the fifth bitmap data corresponding to the second expression; wherein the first expression or the second expression is related to any column of data in the target data; based on the target model, perform a logical operation on the fourth bitmap data and the fifth bitmap data to obtain the sixth bitmap data; and determine the sixth bitmap data as the execution result corresponding to the constraint condition.

[0124] The logic operation in the embodiment of the present invention includes at least one of the following: an AND gate, an OR gate, and a NOT gate.

[0125] In addition, in another possible embodiment, the processing module 1002 may be specifically used to input the expression and the target data into the target model;

[0126] Based on the target model, the SIMD instruction set is used to parallelize the expression and target data to obtain the execution results corresponding to the constraints.

[0127] It should be noted that the data in the column-based database involved in the embodiment of the present invention is stored in columns, and each column separately stores data of at least one attribute.

[0128] In summary, in the embodiment of the present invention, by obtaining a target model corresponding to an expression in a data query request, wherein the expression includes a constraint condition; the expression and the target data in the column-based database are input into the target model, and an execution result corresponding to the constraint condition is obtained. Here, the embodiment of the present invention makes full use of the characteristic that a column-based database can quickly obtain a column of data, reduces the number of writes or outputs in expression calculations, greatly reduces the time for obtaining data, reduces the amount of calculation, improves data processing efficiency, and improves the data query speed of SQL statements.

[0129] In addition, by using the incoming filter mask during the processing, that is, by filtering out some data that does not meet the constraints first, the number of expression operations can be greatly reduced. At the same time, by batch calculation, the SIMD instruction set can be fully utilized for expression calculation, which speeds up the calculation speed.

[0130] Fig.11 A schematic diagram of the structure of a computing device according to an embodiment is shown.

[0131] like Fig.11 1 is a structural diagram of an exemplary hardware architecture of a computing device capable of implementing a data processing method and a data processing apparatus according to an embodiment of the present invention.

[0132] The device may include a processor 1101 and a memory 1102 storing computer program instructions.

[0133] Specifically, the processor 1101 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0134] The memory 1102 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 1102 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 1102 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 1102 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 1102 is a non-volatile solid-state memory. In a particular embodiment, the memory 1102 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0135] The processor 1101 implements any one of the data processing methods in the above embodiments by reading and executing computer program instructions stored in the memory 1102 .

[0136] The transceiver 1103 is mainly used to implement the communication between the devices in the embodiments of the present invention or with other devices.

[0137] In one example, the device may further include a bus 1104. Fig.11 As shown, the processor 1101, the memory 1102 and the transceiver 1103 are connected via a bus 1104 and communicate with each other.

[0138] Bus 1104 includes hardware, software or both.For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industry standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industry standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral control interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In suitable cases, bus 1103 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0139] In addition, an embodiment of the present invention further provides a computer-readable storage medium corresponding to the above-mentioned data processing method. In a possible embodiment, an embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed in a computer, the computer executes the data processing method involved in the embodiment of the present invention.

[0140] It should be clear that the present invention is not limited to the specific configurations and processes described in the above embodiments and shown in the figures. For the convenience and brevity of description, a detailed description of the known methods is omitted here, and the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, which will not be repeated here.

[0141] Those skilled in the art can clearly understand that the method process of the present invention is not limited to the specific steps described and shown. Any technician familiar with the technical field, after understanding the spirit of the present invention, can make various changes, modifications and additions, or equivalent substitutions and change the order of steps within the technical scope disclosed by the present invention. These modifications or substitutions should be included in the protection scope of the present invention.

Claims

1. A data processing method, comprising: Obtain a batch-based target model corresponding to each expression in the data query request; The target model is used to divide the data in the data table into different batches, and each expression processes the data in batches; The expression contains constraints; Inputting the first bitmap data, the expression and the target data in the column-based database into the target model, filtering the target data using the first bitmap data to obtain data satisfying the constraint condition; using a single instruction multiple data stream instruction set, based on the target model and the first bitmap data, processing the data satisfying the constraint condition to obtain an execution result corresponding to the constraint condition; wherein the target data includes all data in at least one column and part of the data in the at least one column; The execution result is outputted.

2. The method according to claim 1, wherein: The target data includes a first column of data and a second column of data, wherein the first column of data and the second column of data have the same number of rows; Based on the target model and the first bitmap data, the data satisfying the constraint condition is processed to obtain an execution result corresponding to the constraint condition, including: Based on the target model, the first column data and the second column data having the same row are added to obtain a third column data; wherein the third column data and the first column data have the same number of rows; The first bitmap data and the third column data are determined as execution results corresponding to the constraint condition.

3. The method according to claim 1, wherein: The target data includes a fourth column of data and constant data, and the fourth column of data and the constant data have the same number of rows; Based on the target model and the first bitmap data, the data satisfying the constraint condition is processed to obtain an execution result corresponding to the constraint condition, including: When the constraint condition is to select data in the fourth column of data that is greater than the constant data, based on the target model, the fourth column of data and the constant data having the same number of rows are compared to obtain an execution result corresponding to the constraint condition; Among them, the execution result includes second bitmap data, the second bitmap data includes visible rows corresponding to rows where the fourth column data with the same row number as the first bitmap data is greater than the constant data, and the second bitmap data has the same number of rows as the fourth column data.

4. The method according to claim 1, wherein: The target model includes a first model and a second model; the target data includes first target data and second target data, the first target data includes a fifth column of data and a sixth column of data, and the fifth column of data and the sixth column of data have the same number of rows; Based on the target model and the first bitmap data, the data satisfying the constraint condition is processed to obtain an execution result corresponding to the constraint condition, including: Based on the first model, the fifth column of data and the sixth column of data having the same row are added to obtain a seventh column of data; wherein the seventh column of data and the fifth column of data have the same number of rows; Inputting the first bitmap data, the seventh column data and the second target data into the second model; wherein the second target data includes the eighth column data, and the eighth column data and the seventh column data have the same number of rows; When the constraint condition is to select data in the seventh column of data that is greater than the eighth column of data, based on the second model, the seventh column of data and the eighth column of data having the same number of rows are compared to obtain an execution result corresponding to the constraint condition; Among them, the execution result includes third bitmap data, and the third bitmap data includes visible rows corresponding to rows where the seventh column data with the same row number as the first bitmap data is greater than the eighth column data, and the third bitmap data has the same number of rows as the seventh column data.

5. The method according to claim 1, wherein: The target data includes a ninth column of data, the first bitmap data includes hidden rows and visible rows, and the first bitmap data and the ninth column of data have the same number of rows; Based on the target model and the first bitmap data, the data satisfying the constraint condition is processed to obtain an execution result corresponding to the constraint condition, including: Based on the target model, using the hidden rows to filter the hidden rows in the ninth column of data, obtaining data in the ninth column of data that has the same row as the visible row; When the expression is data summation, the data of the same row in the visible row are accumulated to obtain accumulated data; The accumulated data and the first bitmap data are determined as execution results corresponding to the constraint condition.

6. The method according to claim 1, wherein: The target data includes a tenth column of data, the first bitmap data includes hidden rows and visible rows, and the first bitmap data and the tenth column of data have the same number of rows; Based on the target model and the first bitmap data, the data satisfying the constraint condition is processed to obtain an execution result corresponding to the constraint condition, including: Based on the target model, using the hidden rows to filter the hidden rows in the tenth column of data, obtaining data in the tenth column of data that has the same row as the visible row; According to the first preset condition in the target model, data in the tenth column of data that is in the same row as the visible row is marked to obtain the marked eleventh column of data; The first bitmap data and the eleventh column of data are determined as execution results corresponding to the constraint condition.

7. The method according to claim 6, wherein: When the target model includes the first preset condition and the second preset condition; the visible row includes a first visible row and a second visible row; According to the first preset condition in the target model, the data in the tenth column of data that has the same row as the visible row is marked to obtain the marked eleventh column of data, including: According to the first preset condition, marking the data in the tenth column of data that has the same row as the first visible row, and adjusting the first visible row to a hidden row; According to the second preset condition, the data in the tenth column of data that has the same row as the second visible row is marked until the third preset condition is met, and the marking is ended to obtain the marked eleventh column of data.

8. The method according to claim 1, wherein: Based on the target model and the first bitmap data, the data satisfying the constraint condition is processed to obtain an execution result corresponding to the constraint condition, including: Determine, according to the first bitmap data, the expression and the target data, fourth bitmap data corresponding to the first expression and fifth bitmap data corresponding to the second expression; wherein the first expression or the second expression is related to any column of data in the target data; Based on the target model, performing a logical operation on the fourth bitmap data and the fifth bitmap data to obtain sixth bitmap data; The sixth bitmap data is determined as an execution result corresponding to the constraint condition.

9. The method according to claim 8, wherein: The logic operation includes at least one of the following: an AND gate, an OR gate, and a NOT gate.

10. The method according to claim 1, wherein: The data in the column-based database is stored in columns, and each column stores data of at least one attribute separately.

11. A data processing device, comprising: An acquisition module, used for acquiring a batch-based target model corresponding to each expression in a data query request; The target model is used to divide the data in the data table into different batches, and each expression processes the data in batches; The expression contains constraints; A processing module, used for inputting the first bitmap data, the expression and the target data in the column-based database into the target model, filtering the target data using the first bitmap data to obtain data satisfying the constraint condition; using a single instruction multiple data stream instruction set, based on the target model and the first bitmap data, processing the data satisfying the constraint condition to obtain an execution result corresponding to the constraint condition; wherein the target data includes all data in at least one column and part of the data in the at least one column; An output module is used to output the execution result.

12. A computing device, wherein: The device comprises at least one processor and a memory, wherein the memory is used to store computer program instructions, and the processor is used to execute the program in the memory to control the computer device to implement any one of the data processing methods in claims 1-10.

13. A computer-readable storage medium having a computer program stored thereon, wherein: If the computer program is executed in a computer, the computer is caused to execute any one of the data processing methods according to claims 1-10.