Data query method and device of columnar database, electronic equipment and storage medium

Abstract

The application relates to the technical field of data processing, and provides a data query method and device of a columnar database, an electronic device and a storage medium, the method comprises the following steps: determining instruction information based on a data query instruction; the instruction information comprises query information and at least one data operation operator; determining query data from the columnar database based on the query information; attaching index codes to each data in the query data based on the storage positions of the data in the columnar database; performing data reorganization on the query data based on the operation characteristics of the data operation operators to obtain reorganized data; performing data processing on the reorganized data based on the data operation operators to obtain processed data; and extracting target data from the columnar database based on the index codes of the data in the processed data. The application can solve the problem that the data organization mode and the index-free mode of the columnar database are difficult to be compatible in heterogeneous computing, and therefore the data query speed can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a data query method, apparatus, electronic device, and storage medium for a columnar database. Background Technology

[0002] With the advent of the artificial intelligence era and the continuous advancement of big data and informatization, the installed capacity of databases is increasing exponentially, and the demand for faster database query speeds is growing daily. Currently, most databases rely on the Central Processing Unit (CPU) on servers to process user data. However, CPUs place a significant burden on processes and memory when handling computationally intensive tasks, affecting the task scheduling and processing speed of other server processes. Therefore, using Field-Programmable Gate Arrays (FPGAs) and Graphics Processing Units (GPUs) as coprocessors to share the computationally intensive tasks of databases has become an emerging technological direction. As data volumes continue to expand, column-oriented databases are gaining popularity to reduce system input / output (IO). However, compared to row-oriented databases, the data organization mode and index-free mode of column-oriented databases are difficult to reconcile in heterogeneous computing, resulting in slow data query speeds. Summary of the Invention

[0003] This application provides a data query method, apparatus, electronic device, and storage medium for columnar databases to solve the problem of slow speed when performing data queries in columnar databases.

[0004] In a first aspect, embodiments of this application provide a data query method for a columnar database, including:

[0005] The instruction information is determined based on the data query instruction; the instruction information includes query information and at least one data operation operator.

[0006] Based on the query information, the query data is determined from the columnar database;

[0007] Based on the storage location of each piece of data in the query data in the columnar database, an index code is added to each piece of data in the query data;

[0008] Based on the operational characteristics of each of the data manipulation operators, the query data is reorganized to obtain reorganized data;

[0009] The recombined data is processed based on the data operation operators described above to obtain processed data;

[0010] Based on the index code of each data in the processed data, the target data is extracted from the columnar database.

[0011] In one embodiment, extracting target data from the columnar database based on the index encoding of each data item in the processed data includes:

[0012] Determine the order information of the index codes of each data in the processed data;

[0013] According to the order information, extract the data corresponding to each index code in the processed data from the columnar database;

[0014] Target data is generated based on the extracted data.

[0015] In one embodiment, the step of reorganizing the query data based on the operational characteristics of each of the data operation operators to obtain reorganized data includes:

[0016] Determine the operational characteristics of each of the data manipulation operators;

[0017] Based on the operational characteristics of each of the data manipulation operators, the query data is reorganized to obtain reorganized data.

[0018] In one embodiment, determining the instruction information based on the data query instruction includes:

[0019] The data query command is parsed to obtain parsing information; the parsing information includes query information and operator information.

[0020] At least one data operation operator is determined from the operator information;

[0021] The at least one data operation operator is transformed to obtain the at least one transformed data operation operator.

[0022] In one embodiment, determining the query data from the columnar database based on the query information includes:

[0023] Using the query information as the retrieval criteria, data is retrieved in the columnar database;

[0024] Query data is generated based on the data in the columnar database that matches the query information.

[0025] In one embodiment, after determining the query data from the columnar database based on the query information, the method further includes at least one of the following:

[0026] The queried data is encrypted.

[0027] The queried data is then compressed.

[0028] In one embodiment, after determining the instruction information based on the data query instruction, and before reorganizing the query data based on the operational characteristics of each of the data operation operators to obtain reorganized data, the method further includes:

[0029] Operator optimization is performed on the at least one data manipulation operator.

[0030] Secondly, embodiments of this application provide a data query device for a columnar database, comprising:

[0031] The first determining module is used to determine instruction information based on a data query instruction; the instruction information includes query information and at least one data operation operator.

[0032] The second determining module is used to determine query data from the columnar database based on the query information;

[0033] An additional module is used to attach an index code to each piece of data in the query data based on the storage location of each piece of data in the columnar database;

[0034] The reorganization module is used to reorganize the query data based on the operational characteristics of each of the data operation operators to obtain reorganized data.

[0035] The processing module is used to process the recombined data based on each of the data operation operators to obtain processed data;

[0036] An extraction module is used to extract target data from the columnar database based on the index encoding of each data in the processed data.

[0037] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the data query method for a columnar database as described in the first aspect.

[0038] Fourthly, embodiments of this application provide a storage medium, which is a computer-readable storage medium including a computer program. When the computer program is executed by a processor, it implements the data query method for the columnar database described in the first aspect.

[0039] The data query method, apparatus, electronic device, and storage medium for column-oriented databases provided in this application determine instruction information including query information and at least one data operation operator through a data query instruction; then, query data can be determined from the column-oriented database based on the query information, and an index code is added to each data in the query data according to the storage location of each data in the column-oriented database; further, the query data is reorganized according to the operation characteristics of each data operation operator, thereby overcoming the dependence on the original data organization mode of the column-oriented database, improving the data processing speed, and obtaining reorganized data; further, the reorganized data can be processed based on each data operation operator to obtain processed data; and then, target data can be extracted from the column-oriented database by indexing according to the index code of each data in the processed data, thereby solving the problem of incompatibility between the data organization mode of the column-oriented database and the indexless mode in heterogeneous computing, and improving the data query speed. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating the data query method for a columnar database provided in an embodiment of this application;

[0042] Figure 2 This is a schematic diagram of the CPU+X heterogeneous architecture provided in an embodiment of this application;

[0043] Figure 3 This is a schematic diagram of the index encoding and data reorganization of column data provided in an embodiment of this application;

[0044] Figure 4 This is a schematic diagram illustrating the working principle of query acceleration provided in the embodiments of this application;

[0045] Figure 5 This is a schematic diagram of the functional modules of an embodiment of the data query device for the database listed in this application;

[0046] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. In the description of this specification, the reference to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., means that the specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the embodiments of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and combine the different embodiments or examples and the features of different embodiments or examples described in this specification. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] The data query method, apparatus, electronic device and storage medium of the columnar database provided in this application will be described in detail below with reference to the embodiments.

[0049] Figure 1 This is a flowchart illustrating a data query method for a columnar database provided in an embodiment of this application. (Refer to...) Figure 1 This application provides a data query method for a columnar database, which may include:

[0050] Step 100: Determine instruction information based on data query instructions;

[0051] It should be noted that the execution subject of the data query method for the columnar database provided in this application embodiment can be a server, computer equipment, such as mobile phone, tablet computer, laptop computer, handheld computer, vehicle electronic equipment, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc.

[0052] Furthermore, the executing entity of this application may include a CPU+X heterogeneous architecture, where X can be an FPGA or GPU, etc. The CPU+X heterogeneous architecture of this application may be connected to a data query device for a columnar database, thereby enabling the data query method for the columnar database of this application to be completed by controlling the data query device.

[0053] Figure 2This is a schematic diagram of a CPU+X heterogeneous architecture provided in an embodiment of this application. (Refer to...) Figure 2 In one embodiment, the CPU may include a terminal application layer, an instruction parsing layer, and a task management layer, and X may include a query acceleration layer.

[0054] The terminal application layer provides users with a graphical or terminal-based application interface, and users can set corresponding data query commands for the columnar database in the terminal application layer.

[0055] In this application's column-oriented database, data is stored column-wise, with each column stored separately, and the data itself serves as the index.

[0056] The instruction parsing layer consists of three parts: instruction parsing, instruction decomposition, and operator optimization. Instruction decomposition breaks down a data query instruction into several data operation operators, which work together to complete the function of the instruction. Operator optimization optimizes the data operation operators derived from the instruction decomposition. Instruction parsing translates the decomposed or optimized operators into low-level code that the CPU can understand.

[0057] The task management layer consists of five parts: task scheduling, driver management, data encryption, data compression, and data transmission. Task scheduling comprehensively manages the processes of operator execution, data encryption, data compression, data transmission, and driver management, ensuring the orderly execution of operator operations and data processing. Driver management manages the scheduling of drivers for essential components such as the Peripheral Component Interconnect Express (PCIE) and XRT, ensuring normal communication and management between the database host and the accelerator card. Data encryption, data compression, and data transmission perform optional data processing and transmission on the data involved in operator operations. These three data transmission parts ensure data transmission between the CPU and the accelerator card, while data encryption and compression are optional functions. Users can compress and encrypt data during transmission as needed to ensure data security and further improve transmission speed. XRT is a software toolkit developed by Xilinx, primarily used to execute applications and algorithms on Xilinx accelerator cards, and provides optimization and management functions for specific hardware architectures to achieve high-performance computing acceleration.

[0058] The query acceleration layer consists of three parts: index encoding, data reconstruction, and query acceleration. Index encoding adds index codes to the column data of the column-store database so that the query results can be extracted based on the output index codes after the final results are output. Data reconstruction reconstructs the column data with added index codes into a data pattern that can be parsed by subsequent acceleration units. Query acceleration is used to parse the reconstructed data and accelerate the operation according to the operators decomposed by the instructions.

[0059] After receiving a data query instruction sent by a user based on the terminal application layer, this application can parse the data query instruction through the CPU's instruction parsing layer to obtain instruction information including query information and at least one data operation operator.

[0060] The query information can be the query conditions; data operation operators can include, but are not limited to, sorting, aggregation, joining, and filtering.

[0061] Step 200: Determine the query data from the columnar database based on the query information;

[0062] After obtaining the query information, the task management layer in the CPU can use the query information as a retrieval condition to retrieve data in the columnar database, and the retrieved data will form the query data.

[0063] In one embodiment, determining query data from a columnar database based on query information includes:

[0064] Step 201: Using the query information as the retrieval criteria, perform data retrieval in the columnar database;

[0065] Step 202: Generate query data based on the data in the columnar database that matches the query information.

[0066] This application can use the task management layer in the CPU to retrieve data from the columnar database by using the query information as the retrieval condition, thereby checking whether data matching the query information exists in the columnar database.

[0067] Furthermore, if data matching the query information exists in the columnar database, then all data matching the query information will be used together as the query data.

[0068] Step 300: Based on the storage location of each piece of data in the columnar database, add an index code to each piece of data in the query data;

[0069] It should be noted that in this application, an index code can be set in advance for each storage location in the columnar database to facilitate subsequent data indexing.

[0070] Therefore, after obtaining the query data, this application can attach an index code to each piece of data in the query data by means of the association between the index code and the storage location in the columnar database.

[0071] Step 400: Based on the operational characteristics of each data operation operator, the query data is reorganized to obtain reorganized data;

[0072] Furthermore, the query acceleration layer in X can be used to reorganize the query data based on the operational characteristics of each data operation operator to obtain reorganized data.

[0073] Operational characteristics refer to the specific operational methods, rules, or features possessed by each operator during data processing. These characteristics describe the behavior and properties of an operator when performing a specific operation.

[0074] For example, if the data manipulation operator is a sorting operator, the operational characteristics of the sorting operator can be determined, and the query data can be reorganized according to the operational characteristics of the sorting operator to obtain the reorganized data.

[0075] For example, if the data manipulation operator is an aggregation operator, the operational characteristics of the aggregation operator can be determined, and the query data can be reorganized according to the operational characteristics of the aggregation operator. After the data reorganization is completed, the reorganized data is obtained.

[0076] Step 500: Perform data processing on the recombined data based on each data operation operator to obtain processed data;

[0077] Furthermore, this application can query the acceleration layer to perform corresponding operations on the recombined data according to each data operation operator, and obtain the processed data after the operation is completed.

[0078] For example, if the data operation operator is a sorting operator, then the reorganized data is sorted, and the processed data is obtained after the sorting is completed.

[0079] For example, if the data manipulation operator is a join operator, then the recombined data is joined, and the processed data is obtained after the join is completed.

[0080] Step 600: Extract target data from columnar database based on the index encoding of each data in the processing data.

[0081] After obtaining the processed data, this application can determine the index code attached to each data in the processed data, and extract data from the columnar database according to the order of the index codes in the processed data to form the target data.

[0082] The data query method for column-oriented databases provided in this application determines instruction information, including query information and at least one data operation operator, through a data query instruction. Then, query data can be determined from the column-oriented database based on the query information, and an index code is added to each piece of data in the query data according to its storage location in the column-oriented database. Further, the query data is reorganized according to the operational characteristics of each data operation operator, thereby overcoming the dependence on the original data organization mode of the column-oriented database, improving data processing speed, and obtaining reorganized data. Further, data processing can be performed on the reorganized data based on each data operation operator to obtain processed data. Finally, target data can be extracted from the column-oriented database using an index based on the index code of each piece of data in the processed data, thereby solving the problem of incompatibility between the data organization mode and the indexless mode of the column-oriented database in heterogeneous computing and improving data query speed.

[0083] This application addresses the data organization model of column-oriented databases by using a CPU+X heterogeneous architecture to accelerate the query process. X (FPGA, GPU, etc.) parses and indexes the column-oriented data according to different query processes, adds index codes to the column data, and reorganizes the column data according to the operational characteristics of different operators. This overcomes the dependency of the query acceleration layer (also known as the acceleration unit) on the data organization model of the column-oriented database. Subsequent acceleration units further accelerate the query of decoded data, offloading the query process of the column-oriented database to heterogeneous computing cards, thereby improving the query speed of the column-oriented database.

[0084] In one embodiment, determining instruction information based on a data query instruction includes:

[0085] Step 101: Parse the data query command to obtain parsing information; the parsing information includes query information and operator information.

[0086] Step 102: Determine at least one data operation operator from the operator information;

[0087] Step 103: Perform data transformation on at least one data operation operator to obtain at least one transformed data operation operator.

[0088] This application can parse data query instructions through the CPU's instruction parsing layer to obtain parsed information including query information and operator information.

[0089] The operator information can consist of one or more data operation operators.

[0090] Therefore, this application can extract at least one data operation operator from the operator information.

[0091] In one embodiment, after parsing the data query instruction, this application can also obtain instruction information containing only query information. If the instruction information containing only query information is obtained, the corresponding data can be directly retrieved from the columnar database based on the query information and output.

[0092] Furthermore, this application can also use an instruction parsing layer to convert each acquired data operation operator into low-level code that the CPU can understand. Specifically, it can convert them into a specified code format, thereby completing the conversion of data operation operators and obtaining the converted data operation operators.

[0093] This application parses data query instructions to obtain at least one data operation operator, which facilitates the reorganization of the obtained query data according to the operation characteristics of each data operation operator. This overcomes the dependence on the original data organization mode of columnar database and improves data processing speed.

[0094] In one embodiment, the query data is reorganized based on the operational characteristics of each data operation operator to obtain reorganized data, including:

[0095] Step 401: Determine the operational characteristics of each data operation operator;

[0096] Step 402: Based on the operational characteristics of each data operation operator, the query data is reorganized to obtain reorganized data.

[0097] After completing the index encoding, this application needs to reorganize the column data in the query data according to the operation characteristics of the data operation operators because different query operations and operation conditions of column-stored databases have different requirements for data format.

[0098] Taking the sorting operator as an example, for the data query instruction "SELECT * FROM TABLE ORDERBY A, B, C", the CPU inputs three columns of data (c0i, c1i, c2i) to the query acceleration layer. For this operation, the query acceleration layer needs to first sort column A, then sort column B if columns A are the same, and finally sort column C if columns B are the same. However, before this, according to the operation characteristics of the sorting operator, the three columns of data with added index codes need to be reorganized into the form (c1i, c0i, c1i, c2i). After reorganization, the reorganized data is obtained.

[0099] If the data manipulation operator is an aggregation operator, for the data query instruction "SELECT sum(A), AVG(A), MAX(C) FROM TABLE GROUP BY A, B, C", the CPU inputs three columns of data (c0i, c1i, c2i) to the query acceleration layer. For this operation, the query acceleration layer needs to group the data according to A, B, and C, and calculate the sum of A, the average of A, and the maximum value of C. However, before this, according to the operational characteristics of the aggregation operator, the three columns of data with added index codes need to be reorganized into the form (Idi, c0i, c1i, c2i, c0i, c0i, c2i) to ensure that the final output matches the data query instruction. After reorganization, the reorganized data is obtained.

[0100] This application can reorganize the queried data according to the operational characteristics of each data operation operator, thereby overcoming the dependence on the original data organization mode of the column-stored database, improving the data processing speed, and obtaining reorganized data; further, based on each data operation operator, the reorganized data is processed to obtain processed data; then, the target data can be extracted from the column-stored database by means of an index, according to the index code of each data in the processed data, thereby solving the problem of the incompatibility between the data organization mode of the column-stored database and the indexless mode in heterogeneous computing, and improving the data query speed.

[0101] In one embodiment, extracting target data from a columnar database based on the index encoding of each data item in the processed data includes:

[0102] Step 601: Determine the order information of the index codes of each data in the processed data;

[0103] Step 602: Extract the data corresponding to each index code from the columnar database according to the sequence information;

[0104] Step 603: Generate target data based on the extracted data.

[0105] This application can determine the order information of the index codes of each data in the processing data through the task management layer in the CPU.

[0106] Furthermore, data corresponding to each index code in the processed data can be extracted from the columnar database according to the determined sequence information.

[0107] Furthermore, the target data is composed of data extracted sequentially and sorted according to the extraction order, facilitating its output for user viewing. It should be noted that if the processed data also includes processing results, these results can be output along with the target data.

[0108] This application can extract target data from a columnar database by using an index to extract data based on the index codes of each data in the processed data. This solves the problem of incompatibility between the data organization mode of a columnar database and the indexless mode in heterogeneous computing, and improves the data query speed.

[0109] In one embodiment, after determining the query data from the columnar database based on the query information, the method further includes at least one of the following:

[0110] Step A: Encrypt the queried data;

[0111] Step B involves compressing the queried data.

[0112] In this application, data encryption and data compression are optional functions. Users can compress and encrypt the data transmission process according to their needs, which can be achieved by checking the boxes in the display interface of the terminal application layer.

[0113] Therefore, when the user selects the data encryption option, this application can encrypt the query data through the task management layer according to the specified data encryption algorithm to obtain the encrypted query data.

[0114] When the user selects the data compression option, the query data can be compressed using the specified data compression algorithm through the task management layer to obtain the compressed query data.

[0115] This application ensures data security and further improves transmission speed by encrypting and compressing the queried data.

[0116] In one embodiment, after determining the instruction information based on the data query instruction, and before reorganizing the query data based on the operational characteristics of each data operation operator to obtain the reorganized data, the method further includes:

[0117] Step C involves optimizing at least one data manipulation operator.

[0118] After determining the instruction information based on the data query instruction, and before reorganizing the query data based on the operational characteristics of each data operation operator to obtain the reorganized data, this application can also optimize at least one data operation operator.

[0119] On the one hand, to address the incoherence between data operation operators, parallelization techniques can be used to execute some data operation operators in parallel, thereby improving overall operating efficiency.

[0120] For example, for a data query instruction that includes multiple data operation operators, some of the independent data operation operators can be executed in parallel using parallelization techniques, thereby reducing the overall execution time.

[0121] On the other hand, for each data operation operator, the specific operation can be optimized to improve its execution efficiency.

[0122] For example, for sorting operators, more efficient sorting algorithms (such as quicksort) can be used to replace traditional algorithms (such as bubble sort), thereby reducing the time complexity of sorting.

[0123] In addition, for aggregation operators, aggregation algorithms that are more suitable for data characteristics can be selected to reduce unnecessary calculation steps and lower computational costs.

[0124] This application optimizes data manipulation operators to maximize the efficiency of operation between operators and the acceleration of the query process, thereby improving data query speed.

[0125] Figure 3 This diagram illustrates the indexing and data reorganization of column data provided in an embodiment of this application. (Refer to...) Figure 3 In one embodiment, when performing heterogeneous acceleration, such as Figure 3 As shown, the CPU will send the column data involved in this query from the host memory to the memory of the heterogeneous acceleration card (i.e., the query acceleration layer) one by one (①). This storage mode and data organization mode cannot be directly used by the subsequent acceleration unit. In order to ensure that the column database obtains the calculation results, this application adds index encoding to the column data (②). After the calculation acceleration is completed, it is only necessary to return the result index encoding (Id0, Id1, ..., Idn) to the CPU. The CPU can extract the calculation results through the index encoding and complete the query acceleration.

[0126] After completing the index encoding, this application reorganizes the data according to the operation operators (i.e., data manipulation operators) (③). Different database query operations and operation conditions have different requirements for data format. This application reorganizes the column data according to the operation operators. Taking the sorting operator as an example, for the query operation "SELECT * FROM TABLE ORDER BY A, B, C", the host memory inputs three columns of column data into the accelerator card's memory. To complete the sorting operation, the accelerator card needs to sort column A first, then sort column B if columns A are the same, and then sort column C if columns B are the same. For this operation, the three columns of column data with index codes are reorganized, such as... Figure 3 As shown, after the reorganization is completed, the accelerator card can perform calculations on the reorganized data and output the index code of the final sorting result.

[0127] After accelerating the query, the accelerator card returns the index code of the calculation result to the CPU. The CPU then retrieves the data from the host memory or hard disk according to the index code, thus completing the query operation.

[0128] Figure 4 A schematic diagram illustrating the working principle of query acceleration provided in this application embodiment. (Refer to...) Figure 4 In one embodiment, in the CPU+X heterogeneous architecture of this application, the CPU can be located in a database server, where the CPU can perform read and write operations on the host memory of the database server; X (FPGA, GPU, etc.) can be located in a data acceleration card, and X can include sorting acceleration cores, aggregation acceleration cores, filtering acceleration cores, and connection acceleration cores for query acceleration. After receiving the column data obtained and transmitted by the CPU from the host memory, the data can be processed by the corresponding acceleration cores after indexing and data reorganization, and the index results can be returned in DDR4 memory type, where DDR4 is a computer memory type.

[0129] Furthermore, this application also provides a data query device for a columnar database.

[0130] Reference Figure 5 , Figure 5 This is a schematic diagram of the functional modules of a data query device embodiment for storing databases in this application.

[0131] The data query device for the columnar database includes:

[0132] The first determining module 510 is used to determine instruction information based on a data query instruction; the instruction information includes query information and at least one data operation operator.

[0133] The second determining module 520 is used to determine query data from the columnar database based on the query information;

[0134] Additional module 530 is used to add an index code to each piece of data in the query data based on the storage location of each piece of data in the columnar database;

[0135] The recombination module 540 is used to reorganize the query data based on the operational characteristics of each of the data operation operators to obtain reorganized data.

[0136] Processing module 550 is used to process the recombined data based on each of the data operation operators to obtain processed data;

[0137] Extraction module 560 is used to extract target data from the columnar database based on the index encoding of each data in the processed data.

[0138] The data query device for a columnar database provided in this application determines instruction information, including query information and at least one data operation operator, through a data query instruction. Then, it can determine query data from the columnar database based on the query information, and add an index code to each piece of data in the query data according to the storage location of each piece of data in the columnar database. Further, it reorganizes the query data according to the operational characteristics of each data operation operator, thereby overcoming the dependence on the original data organization mode of the columnar database, improving data processing speed, and obtaining reorganized data. Further, it can process the reorganized data based on each data operation operator to obtain processed data. Finally, it can extract target data from the columnar database using an index based on the index code of each piece of data in the processed data, thereby solving the problem of incompatibility between the data organization mode and the indexless mode of the columnar database in heterogeneous computing and improving data query speed.

[0139] In one embodiment, the first determining module 510 is specifically used for:

[0140] The data query command is parsed to obtain parsing information; the parsing information includes query information and operator information.

[0141] At least one data operation operator is determined from the operator information;

[0142] The at least one data operation operator is transformed to obtain the at least one transformed data operation operator.

[0143] In one embodiment, the second determining module 520 is specifically used for:

[0144] Using the query information as the retrieval criteria, data is retrieved in the columnar database;

[0145] Query data is generated based on the data in the columnar database that matches the query information.

[0146] In one embodiment, the second determining module 520 further includes a processing unit, the processing unit being used for:

[0147] The queried data is encrypted.

[0148] The queried data is then compressed.

[0149] In one embodiment, the second determining module 520 further includes an optimization unit, the optimization unit being used for:

[0150] Operator optimization is performed on the at least one data manipulation operator.

[0151] In one embodiment, the recombination module 540 is specifically used for:

[0152] Determine the operational characteristics of each of the data manipulation operators;

[0153] Based on the operational characteristics of each of the data manipulation operators, the query data is reorganized to obtain reorganized data.

[0154] In one embodiment, the extraction module 560 is specifically used for:

[0155] Determine the order information of the index codes of each data in the processed data;

[0156] According to the order information, extract the data corresponding to each index code in the processed data from the columnar database;

[0157] Target data is generated based on the extracted data.

[0158] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call a computer program in the memory 630 to execute the steps of a data query method for a columnar database, such as including:

[0159] The instruction information is determined based on the data query instruction; the instruction information includes query information and at least one data operation operator.

[0160] Based on the query information, the query data is determined from the columnar database;

[0161] Based on the storage location of each piece of data in the query data in the columnar database, an index code is added to each piece of data in the query data;

[0162] Based on the operational characteristics of each of the data manipulation operators, the query data is reorganized to obtain reorganized data;

[0163] The recombined data is processed based on the data operation operators described above to obtain processed data;

[0164] Based on the index code of each data in the processed data, the target data is extracted from the columnar database.

[0165] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0166] On the other hand, embodiments of this application also provide a storage medium, which is a computer-readable storage medium storing a computer program. The computer program is used to cause a processor to execute the steps of the methods provided in the above embodiments, including, for example:

[0167] The instruction information is determined based on the data query instruction; the instruction information includes query information and at least one data operation operator.

[0168] Based on the query information, the query data is determined from the columnar database;

[0169] Based on the storage location of each piece of data in the query data in the columnar database, an index code is added to each piece of data in the query data;

[0170] Based on the operational characteristics of each of the data manipulation operators, the query data is reorganized to obtain reorganized data;

[0171] The recombined data is processed based on the data operation operators described above to obtain processed data;

[0172] Based on the index code of each data in the processed data, the target data is extracted from the columnar database.

[0173] The computer-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic storage (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical storage (e.g., CD, DVD, BD, HVD), and semiconductor storage (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0174] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0175] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data query method of a columnar database, characterized by, include: Determine instruction information based on data query instructions; The instruction information includes query information and at least one data operation operator; Based on the query information, the query data is determined from the columnar database; Based on the storage location of each piece of data in the query data in the columnar database, an index code is added to each piece of data in the query data; Based on the operational characteristics of each of the data manipulation operators, the query data is reorganized to obtain reorganized data; The recombined data is processed based on the data operation operators described above to obtain processed data; Based on the index code of each data in the processed data, the target data is extracted from the columnar database.

2. The data query method for a columnar database according to claim 1, characterized in that, The step of extracting target data from the columnar database based on the index encoding of each data in the processed data includes: Determine the order information of the index codes of each data in the processed data; According to the order information, extract the data corresponding to each index code in the processed data from the columnar database; Target data is generated based on the extracted data.

3. The data query method for a columnar database according to claim 1, characterized in that, The process of reorganizing the query data based on the operational characteristics of each of the data operation operators to obtain reorganized data includes: Determine the operational characteristics of each of the data manipulation operators; Based on the operational characteristics of each of the data manipulation operators, the query data is reorganized to obtain reorganized data.

4. The data query method for a columnar database according to claim 1, characterized in that, The process of determining instruction information based on data query instructions includes: The data query command is parsed to obtain parsing information; the parsing information includes query information and operator information. At least one data operation operator is determined from the operator information; The at least one data operation operator is transformed to obtain the at least one transformed data operation operator.

5. The data query method for a columnar database according to claim 1, characterized in that, The step of determining the query data from the columnar database based on the query information includes: Using the query information as the retrieval criteria, data is retrieved in the columnar database; Query data is generated based on the data in the columnar database that matches the query information.

6. The data query method for a columnar database according to claim 1, characterized in that, After determining the query data from the columnar database based on the query information, the process also includes at least one of the following: The queried data is encrypted. The queried data is then compressed.

7. The data query method for a columnar database according to any one of claims 1-6, characterized in that, After determining the instruction information based on the data query instruction, and before reorganizing the query data based on the operational characteristics of each of the data operation operators to obtain the reorganized data, the method further includes: Operator optimization is performed on the at least one data manipulation operator.

8. A data query device for a columnar database, characterized in that, include: The first determining module is used to determine instruction information based on data query instructions; The instruction information includes query information and at least one data operation operator; The second determining module is used to determine query data from the columnar database based on the query information; An additional module is used to attach an index code to each piece of data in the query data based on the storage location of each piece of data in the columnar database; The reorganization module is used to reorganize the query data based on the operational characteristics of each of the data operation operators to obtain reorganized data. The processing module is used to process the recombined data based on each of the data operation operators to obtain processed data; An extraction module is used to extract target data from the columnar database based on the index encoding of each data in the processed data.

9. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the data query method for the columnar database as described in any one of claims 1 to 7.

10. A storage medium, said storage medium being a computer-readable storage medium, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the data query method for the columnar database as described in any one of claims 1 to 7.

Images