Query operation instruction optimization method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose a query operation instruction optimization method and device, electronic equipment and a storage medium, the method comprising: obtaining a query operation instruction, and obtaining corresponding query operation object data according to the query operation instruction, wherein the query operation object data is compressed data carrying corresponding attribute information; determining query operable compressed data corresponding to the query operation object data according to the query operation object data, and obtaining attribute information of the query operable compressed data; and performing hierarchical optimization on the query operation instruction according to the attribute information of the query operable compressed data, to obtain an optimized query operation instruction. The technical solution has strong applicability, avoids a cumbersome data decompression process, greatly saves the storage space of decompressed data, simplifies the data operation process, and improves the data operation performance.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the technical field of query operation control, in particular to a query operation instruction optimization method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the development of data technology, more and more data needs to be stored in a database. In order to save data storage space, it is usually necessary to compress the data to be stored before storage, for example, for a HiStore columnar database, a suitable compression algorithm can achieve an average compression ratio of more than 10:1, which can greatly save the data storage space. However, when subsequent data operations such as data query are performed, the compressed data cannot be directly operated, for example, if the original execution engine uses the original data type consistent with the logical execution plan during execution, the compressed data needs to be completely decompressed and restored to the original data before the query operation is performed, so that the tedious data decompression process will greatly reduce the query performance of the engine.

[0003] For the above problems, the following processing methods can be used: 1, data eager decompression, that is, decompression is performed when the data is imported into the main memory, this processing method can limit the code changes caused by the compressed storage manager, but it cannot avoid the decompression of the data, so the execution process of the data operation has not been substantially optimized, and since the data stored in the memory is uncompressed original data, the memory occupation has not been optimized; 2, lazy decompression, that is, the data is kept in a compressed state as much as possible during the entire query operation execution process, and is only decompressed when necessary, such as before physical operator operation, but this processing method is only applicable to some compression methods that provide complete mapping, that is, the decompressed data can be obtained through complete mapping, and additional mapping operations need to be added, which will slow down the execution speed and may increase the intermediate result size; 3, transient decompression, that is, by adjusting the standard relational operator, the compression attribute of the data is not changed before and after the execution operation, and the data is only decompressed within the current operator operation, but this processing method is basically only applicable to relational operators, and the application occasions are very limited. SUMMARY

[0004] The embodiment of the present application provides a query operation instruction optimization method and device, electronic equipment and storage medium.

[0005] In a first aspect, the embodiment of the present application provides a query operation instruction optimization method.

[0006] Specifically, the query operation instruction optimization method comprises the following steps.

[0007] Obtaining a query operation instruction, and obtaining corresponding query operation object data according to the query operation instruction, wherein the query operation object data is compressed data carrying corresponding attribute information.

[0008] Determining query operable compressed data corresponding to the query operation object data according to the query operation object data, and obtaining attribute information of the query operable compressed data.

[0009] Carrying out hierarchical optimization on the query operation instruction according to the attribute information of the query operable compressed data, to obtain an optimized query operation instruction.

[0010] With reference to the first aspect, in a first implementation manner of the first aspect, the compressed data is stored column by column in units of row groups, and the compressed data comprises compressed data packets for storing compressed data and compressed data information packets for storing attribute information of the compressed data.

[0011] With reference to the first aspect and the first implementation manner of the first aspect, in a second implementation manner of the first aspect, the determining query operable compressed data corresponding to the query operation object data according to the query operation object data, and the obtaining attribute information of the query operable compressed data, comprise the following steps.

[0012] When the query operation object data is operable compressed data, the query operation object data is determined as the query operable compressed data corresponding thereto.

[0013] When the query operation object data is inoperable compressed data, the inoperable compressed data is decompressed until operable compressed data is obtained, and the obtained operable compressed data is determined as the query operable compressed data corresponding to the query operation object data.

[0014] Obtaining attribute information of the query operable compressed data, wherein the attribute information of the query operable compressed data comprises one or more of the following information: statistical information, compression information, and storage information.

[0015] With reference to the first aspect, the first implementation manner of the first aspect, and the second implementation manner of the first aspect, in a third implementation manner of the first aspect, the carrying out hierarchical optimization on the query operation instruction according to the attribute information of the query operable compressed data, to obtain an optimized query operation instruction, comprises the following steps.

[0016] Determining a query operation expression and a query operation operator corresponding to the query operation instruction according to the query operation instruction.

[0017] obtaining first attribute information of the query operable compressed data, performing primary optimization on the query operation instruction according to the first attribute information, obtaining a corresponding primary optimized query operation expression and a primary optimized query operation operator;

[0018] obtaining second attribute information of the query operable compressed data, performing secondary optimization on the primary optimized query operation expression according to the second attribute information, obtaining a corresponding secondary optimized query operation expression and a secondary optimized query operation operator, until obtaining a final optimized query operation expression and a final optimized query operation operator, and obtaining the optimized query operation instruction according to the final optimized query operation expression and the final optimized query operation operator.

[0019] With reference to the first implementation manner of the first aspect, the second implementation manner of the first aspect, the third implementation manner of the first aspect, and the fourth implementation manner of the first aspect, in a fifth implementation manner of the first aspect, the method further includes:

[0020] performing operation on the query operable compressed data according to the optimized query operation instruction.

[0021] With reference to the first aspect, the first implementation manner of the first aspect, the second implementation manner of the first aspect, the third implementation manner of the first aspect, the fourth implementation manner of the first aspect, and the fifth implementation manner of the first aspect, in a sixth implementation manner of the first aspect, the performing operation on the query operable compressed data according to the optimized query operation instruction includes:

[0022] determining target query operable compressed data in the query operable compressed data according to the optimized query operation instruction;

[0023] performing operation on the target query operable compressed data according to the optimized query operation instruction.

[0024] In a second aspect, a query operation instruction optimization method is provided in the embodiments of the present application.

[0025] Specifically, the query operation instruction optimization method includes:

[0026] an obtaining module configured to obtain a query operation instruction, and obtain corresponding query operation object data according to the query operation instruction, wherein the query operation object data is compressed data carrying corresponding attribute information;

[0027] a determining module configured to determine query operable compressed data corresponding to the query operation object data according to the query operation object data, and obtain attribute information of the query operable compressed data;

[0028] The optimization module is configured to perform hierarchical optimization on the query operation instruction according to attribute information of the query operable compressed data, to obtain an optimized query operation instruction.

[0029] With reference to the second aspect, in a first implementation manner of the second aspect, the compressed data is stored column by column in units of row groups, and the compressed data includes compressed data packets for storing compressed data and compressed data information packets for storing attribute information of the compressed data.

[0030] With reference to the second aspect and the first implementation manner of the second aspect, in a second implementation manner of the second aspect, the determination module includes:

[0031] The first determination sub-module is configured to, when the query operation object data is operable compressed data, determine the query operation object data as query operable compressed data corresponding to the query operation object data.

[0032] The second determination sub-module is configured to, when the query operation object data is inoperable compressed data, decompress the inoperable compressed data until operable compressed data is obtained, and determine the obtained operable compressed data as query operable compressed data corresponding to the query operation object data.

[0033] The acquisition sub-module is configured to acquire attribute information of the query operable compressed data, where the attribute information of the query operable compressed data includes one or more of the following information: statistical information, compression information, and storage information.

[0034] With reference to the second aspect, the first implementation manner of the second aspect, and the second implementation manner of the second aspect, in a third implementation manner of the second aspect, the optimization module includes:

[0035] The third determination sub-module is configured to determine, according to the query operation instruction, a query operation expression and a query operation operator corresponding to the query operation instruction.

[0036] The first optimization sub-module is configured to acquire first attribute information of the query operable compressed data, perform first-level optimization on the query operation instruction according to the first attribute information, to obtain a first-level optimized query operation expression and a first-level optimized query operation operator.

[0037] The second optimization submodule is configured to obtain second attribute information of the query operable compressed data, perform secondary optimization on the first-level optimized query operation expression according to the second attribute information, obtain a corresponding secondary optimized query operation expression and a secondary optimized query operation operator, and perform tertiary optimization until a final optimized query operation expression and a final optimized query operation operator are obtained, and obtain the optimized query operation instruction according to the final optimized query operation expression and the final optimized query operation operator.

[0038] With reference to the first implementation manner of the second aspect, the second implementation manner of the second aspect, the third implementation manner of the second aspect, and the fourth implementation manner of the second aspect, in a fifth implementation manner of the second aspect, the execution module includes:

[0039] The execution module is configured to perform operation on the query operable compressed data according to the optimized query operation instruction.

[0040] With reference to the first implementation manner of the second aspect, the second implementation manner of the second aspect, the third implementation manner of the second aspect, the fourth implementation manner of the second aspect, and the fifth implementation manner of the second aspect, in a sixth implementation manner of the second aspect, the execution module includes:

[0041] The fourth determination submodule is configured to determine target query operable compressed data in the query operable compressed data according to the optimized query operation instruction.

[0042] The execution submodule is configured to perform operation on the target query operable compressed data according to the optimized query operation instruction.

[0043] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, the memory is used to store one or more computer instructions supporting a query operation instruction optimization device to execute the query operation instruction optimization method described above, and the processor is configured to execute the computer instructions stored in the memory. The query operation instruction optimization device can also include a communication interface, which is used for communication between the query operation instruction optimization device and other devices or communication networks.

[0044] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which is used to store computer instructions for a query operation instruction optimization device, and includes computer instructions involved in the query operation instruction optimization method for the query operation instruction optimization device.

[0045] The technical scheme provided by the embodiment of the present application can include the following beneficial effects:

[0046] The technical solution above uses attribute information of the query operation object data to perform hierarchical optimization on the query operation instruction, obtains an optimized query operation instruction, and then directly operates on the compressed data using the optimized query operation instruction. The technical solution has strong applicability, avoids a complicated data decompression process, greatly saves storage space of the decompressed data, simplifies a data operation process, and improves data operation performance.

[0047] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory and are not restrictive of the embodiments of the application. BRIEF DESCRIPTION OF DRAWINGS

[0048] Other features, objects, and advantages of the embodiments of the application will become more apparent from the following detailed description of non-limiting embodiments with reference to the drawings. In the drawings:

[0049] Figure 1 A flowchart of a query operation instruction optimization method according to an embodiment of the application is shown;

[0050] Figure 2 A schematic diagram of column-by-column storage of compressed data according to an embodiment of the application is shown;

[0051] Figure 3 A flowchart of data compression and compressed data query according to an embodiment of the application is shown;

[0052] Figure 4 A flowchart of a query operation instruction optimization method according to another embodiment of the application is shown; Figure 1 A flowchart of step S102 of the query operation instruction optimization method according to the embodiment shown in FIG. 2 is shown;

[0053] Figure 5 A flowchart of data compression and compressed data query according to an embodiment of the application is shown;

[0054] Figure 6 A flowchart of step S103 of the query operation instruction optimization method according to the embodiment shown in FIG. 3 is shown; Figure 1 A flowchart of step S103 of the query operation instruction optimization method according to the embodiment shown in FIG. 3 is shown;

[0055] Figure 7 A flowchart of a query operation instruction optimization method according to another embodiment of the application is shown;

[0056] Figure 8 A flowchart of a query operation instruction optimization method according to another embodiment of the application is shown;

[0057] Figure 9 A flowchart of step S804 of the query operation instruction optimization method according to the embodiment shown in FIG. 8 is shown; Figure 8 A flowchart of step S804 of the query operation instruction optimization method according to the embodiment shown in FIG. 8 is shown;

[0058] Figure 10AThis is a schematic diagram of the logical expression according to Example 1 of the present invention. Figure 10B This is a schematic diagram of database storage according to Example 1 of the present invention. Figure 10C This is a schematic diagram of an existing processing flow according to Example 1 of the present invention. Figure 10D This is a schematic diagram of the processing flow of the present invention according to Example 1;

[0059] Figure 11A This is a schematic diagram of database storage according to Example 2 of the present invention. Figure 11B This is a schematic diagram of an existing processing flow according to Example 2 of the present invention. Figure 11C This is a schematic diagram of the optimized logic expression according to Example 2 of the present invention. Figure 11D This is a schematic diagram of another existing processing flow according to Example 2 of the present invention. Figure 11E This is a schematic diagram of another optimized logic expression according to Example 2 of the present invention;

[0060] Figure 12A This is a schematic diagram of the logical expression according to Example 3 of the present invention. Figure 12B This is a schematic diagram of database storage according to Example 3 of the present invention. Figure 12C This is a schematic diagram of an existing processing flow according to Example 3 of the present invention. Figure 12D This is a schematic diagram of the optimized logic expression according to Example 3 of the present invention;

[0061] Figure 13A This is a schematic diagram of the logical expression according to Example 4 of the present invention. Figure 13B This is a schematic diagram of an existing processing flow according to Example 4 of the present invention. Figure 13C This is a schematic diagram of the optimized logic expression according to Example 4 of the present invention;

[0062] Figure 14A This is a schematic diagram of the logical expression according to Example 5 of the present invention. Figure 14B This is a schematic diagram of an existing processing flow according to Example 5 of the present invention. Figure 14C This is a schematic diagram of the optimized logic expression according to Example 4 of the present invention;

[0063] Figure 15 A structural block diagram of a query operation instruction optimization device according to an embodiment of the present invention is shown;

[0064] Figure 16 Showing according to Figure 15 Structural block diagram of the determination module 1502 of the query operation instruction optimization device in the illustrated embodiment;

[0065] Figure 17 Showing according to Figure 15 Structural block diagram of the optimization module 1503 of the query operation instruction optimization device in the embodiment shown;

[0066] Figure 18Fig. 6 shows a structural block diagram of a query operation instruction optimization device according to another embodiment of the present application;

[0067] Figure 19 Fig. 7 shows a structural block diagram of a query operation instruction optimization device according to Figure 18 Fig. 8 shows a structural block diagram of an execution module 1804 of the query operation instruction optimization device according to the embodiment shown in Fig. 7;

[0068] Figure 20 Fig. 9 shows a structural block diagram of an electronic device according to an embodiment of the present application;

[0069] Figure 21 Fig. 10 is a structural schematic diagram of a computer system suitable for implementing a query operation instruction optimization method according to an embodiment of the present application. DETAILED DESCRIPTION

[0070] Hereinafter, exemplary embodiments of the present application will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. Furthermore, portions irrelevant to the description of the exemplary embodiments are omitted in the accompanying drawings for the sake of clarity.

[0071] In the embodiments of the present application, it should be understood that terms such as "include" or "have" are intended to indicate that there are features, numbers, steps, actions, components, parts or combinations thereof disclosed in the specification, and do not exclude the possibility that one or more other features, numbers, steps, actions, components, parts or combinations thereof exist or are added.

[0072] It should also be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0073] The technical solution provided by the embodiments of the present application classifies and optimizes the query operation instruction by means of the attribute information of the query operation object data, obtains the optimized query operation instruction, and then directly operates the compressed data by using the optimized query operation instruction. The technical solution has strong applicability, avoids the cumbersome data decompression process, greatly saves the storage space of the decompressed data, simplifies the data operation process, and improves the data operation performance.

[0074] Figure 1 Fig. 11 shows a flowchart of a query operation instruction optimization method according to an embodiment of the present application, which is suitable for a query operation instruction optimization server, such as Figure 1 As shown, the query operation instruction optimization method includes the following steps S101-S103:

[0075] In step S101, a query operation instruction is acquired, and corresponding query operation object data is acquired according to the query operation instruction, wherein the query operation object data is compressed data carrying corresponding attribute information.

[0076] In step S102, query operable compressed data corresponding to the query operation object data is determined according to the query operation object data, and attribute information of the query operable compressed data is acquired.

[0077] In step S103, hierarchical optimization is performed on the query operation instruction according to the attribute information of the query operable compressed data, and an optimized query operation instruction is obtained.

[0078] As mentioned above, with the development of data technology, more and more data needs to be stored in a database. In order to save data storage space, it is usually necessary to compress the data to be stored before storage, but when subsequent data query and other data operations are performed, the compressed data cannot be directly operated, and the query operation needs to be performed after the compressed data is completely decompressed and restored to the original data. The tedious data decompression process greatly reduces the query performance of the engine. The existing solutions either do not optimize the execution process of data operation and the occupation of memory in essence, or have very limited application occasions.

[0079] In view of the above problems, in this embodiment, a query operation instruction optimization method is proposed, which performs hierarchical optimization on the query operation instruction by means of attribute information of query operation object data, to obtain an optimized query operation instruction, so as to directly operate on compressed data by using the optimized query operation instruction subsequently. This technical solution has strong applicability, avoids the tedious data decompression process, greatly saves the storage space of decompressed data, simplifies the data operation process, and improves the data operation performance.

[0080] In an embodiment of the present application, the query operation instruction refers to a query instruction for realizing a certain operation purpose, such as a query instruction, a search instruction, etc. For the SQL field, the query operation instruction can be realized as a SQL query operation instruction.

[0081] In an embodiment of the present application, the query operation object data corresponding to the query operation instruction refers to data related to the query operation instruction, which may eventually be the object of query operation. For example, if the query operation instruction is to query a number less than 950 in a database, all data stored in the database can be considered as query operation object data corresponding to the query operation instruction, which may participate in the calculation of the query operation in the subsequent query operation.

[0082] In an embodiment of the present application, the query operation object data is compressed data which has been stored in the database, and the compressed data further carries corresponding attribute information, wherein the attribute information can include one or more of the following information: statistical information of the compressed data, compression information of the compressed data, storage information of the compressed data, etc., and the presence of the attribute information makes it possible for the present application to optimize the query operation instruction, wherein the statistical information can be, for example, maximum value of the data, minimum value of the data, sum value of the data within a certain preset range, data type, etc., the compression information can be, for example, original data type, data storage type, data compression mode, etc., and the storage information can be, for example, storage type, storage location, etc.

[0083] In an embodiment of the present application, the compressed data can be, for example, compressed data stored in a HiStore columnar database in units of row groups, wherein a row group refers to a data unit formed by a plurality of rows in columnar storage, and the row here refers to a data row in the traditional sense in the field of databases, that is, in the HiStore columnar database, data is stored column by column, and in a certain column, the data is further divided into row groups each consisting of a plurality of rows. In addition, the data in a row group can be independently compressed and information- counted in units of row groups, wherein the result of information counting is the statistical information or attribute information of the row group, which is mainly used to describe the characteristics and attributes of the stored data of the row group, and therefore, in this embodiment, the compressed data includes compressed data packets for storing compressed data and compressed data information packets for storing attribute information of the compressed data. Figure 2 A schematic diagram of column-by-column storage of compressed data according to an embodiment of the present application is shown in Figure 2 , which includes column 1, column 2, column 3 and other columns (not shown in Figure 2 ), row group 1, row group 2, row group 3 and other row groups (not shown in Figure 2 ), and in Figure 2 , the intersection of a column and a row group represents a row group unit contained in the column, the black square in each intersection represents the statistical information of the row group, i.e., the compressed data information packet, and the gray square represents the data packet formed by compressing the data of each row contained in the row group, i.e., the compressed data packet.

[0084] In an embodiment of the present application, the operable compressed data refers to data that can be directly queried and the like based on the compressed data, and can also be referred to as interpretable compressed data. For example, the compressed data obtained by using a compression method such as an incremental-based Delta compression, a greatest common divisor-based GCD compression, a Frame-of-reference (FOR) compression, and various Dictionary dictionary compressions are operable compressed data. Similarly, the inoperable compressed data refers to data that cannot be directly queried and the like based on the compressed data, and needs to be decompressed into operable compressed data before being queried and the like, and can also be referred to as uninterpretable compressed data. For example, the compressed data obtained by using a compression method such as Huffman coding, arithmetic coding, and LZ77 / LZ4 algorithms are inoperable compressed data. Figure 3 As shown in FIG. 1, in the prior art, when original data is compressed into compressed data and stored in a disk, the original data is first compressed into operable compressed data with a certain compression rate, and then the operable compressed data is compressed into inoperable compressed data with a higher compression rate for storage. When the compressed data needs to be queried, the inoperable compressed data needs to be decompressed into operable compressed data with a certain readability, and then the operable compressed data needs to be decompressed and restored into the original data for data query operation. Figure 3 As shown in FIG. 1, in the prior art, when original data is compressed into compressed data and stored in a disk, the original data is first compressed into operable compressed data with a certain compression rate, and then the operable compressed data is compressed into inoperable compressed data with a higher compression rate for storage. When the compressed data needs to be queried, the inoperable compressed data needs to be decompressed into operable compressed data with a certain readability, and then the operable compressed data needs to be decompressed and restored into the original data for data query operation.

[0085] In an embodiment of the present application, the query operation instruction is hierarchically optimized according to the attribute information of the query operable compressed data, that is, the query operation instruction is simplified and optimized in different aspects according to the obtained attribute information of the query operable compressed data, to finally obtain an optimized query operation instruction. The specific optimization process of the query operation instruction will be described in detail below.

[0086] As shown in FIG. 1, in the prior art, when original data is compressed into compressed data and stored in a disk, the original data is first compressed into operable compressed data with a certain compression rate, and then the operable compressed data is compressed into inoperable compressed data with a higher compression rate for storage. When the compressed data needs to be queried, the inoperable compressed data needs to be decompressed into operable compressed data with a certain readability, and then the operable compressed data needs to be decompressed and restored into the original data for data query operation. Figure 4 As shown in FIG. 1, in the prior art, when original data is compressed into compressed data and stored in a disk, the original data is first compressed into operable compressed data with a certain compression rate, and then the operable compressed data is compressed into inoperable compressed data with a higher compression rate for storage. When the compressed data needs to be queried, the inoperable compressed data needs to be decompressed into operable compressed data with a certain readability, and then the operable compressed data needs to be decompressed and restored into the original data for data query operation.

[0087] In step S401, when the query operation object data is operable compressed data, the query operation object data is determined as the query operable compressed data corresponding thereto.

[0088] In step S402, when the query operation object data is inoperable compressed data, the inoperable compressed data is decompressed until operable compressed data is obtained, and the decompressed operable compressed data is determined as the query operable compressed data corresponding to the query operation object data.

[0089] In step S403, attribute information of the query operable compressed data is obtained, wherein the attribute information of the query operable compressed data at least includes one or more of the following information: statistical information, compression information, and storage information.

[0090] As mentioned above, in the prior art, when querying the stored compressed data, the inoperable compressed data is first decompressed into operable compressed data with certain readability, and then the operable compressed data is decompressed and restored into original data for data query operation. In order to avoid the tedious data decompression process, save the storage space of the decompressed data, simplify the data operation process, and improve the data operation performance, in the embodiment, in order to be able to directly perform corresponding operations on the compressed data in the future, it is necessary to first determine whether the query operation object data is operable compressed data. If the previously obtained query operation object data is already operable compressed data, the query operation object data is determined as the query operable compressed data corresponding thereto, and subsequent data operations such as query can be directly performed. If the query operation object data is inoperable compressed data, the inoperable compressed data needs to be decompressed until the operable compressed data corresponding thereto is obtained.

[0091] Figure 5 A flowchart of data compression and compressed data query according to an embodiment of the present application is shown in FIG. 1. Figure 5 As shown in FIG. 1, when querying the compressed data, only the inoperable compressed data stored in the database needs to be decompressed into operable compressed data with certain readability, and the operable compressed data does not need to be decompressed and restored into original data, and the data query operation can be performed.

[0092] In an embodiment of the present application, as shown in FIG. 2, the step S103, i.e., the step of performing hierarchical optimization on the query operation instruction according to the attribute information of the query operable compressed data to obtain the optimized query operation instruction, includes the following steps S601-S603. Figure 6

[0093] In step S601, a query operation expression and a query operation operator corresponding to the query operation instruction are determined according to the query operation instruction.

[0094] ​In step S602, first attribute information of the query operable compressed data is acquired, and first-level optimization is performed on the query operation instruction according to the first attribute information, to obtain a corresponding first-level optimized query operation expression and first-level optimized query operation operator.

[0095] In step S603, second attribute information of the query operable compressed data is acquired, and secondary optimization is performed on the first-level optimized query operation expression according to the second attribute information, to obtain a corresponding secondary optimized query operation expression and secondary optimized query operation operator, until the last-level optimized query operation expression and last-level optimized query operation operator are obtained, and the optimized query operation instruction is obtained according to the last-level optimized query operation expression and last-level optimized query operation operator.

[0096] In this embodiment, when performing hierarchical optimization on the query operation instruction according to attribute information of the query operable compressed data, first, the query operation expression and query operation operator corresponding to the query operation instruction are determined according to the query operation instruction; then, first attribute information of the query operable compressed data is acquired, and first-level optimization is performed on the query operation instruction according to the first attribute information, to obtain a corresponding first-level optimized query operation expression and first-level optimized query operation operator; then, second attribute information of the query operable compressed data is acquired, and secondary optimization is performed on the first-level optimized query operation expression according to the second attribute information, to obtain a corresponding secondary optimized query operation expression and secondary optimized query operation operator; and the process is sequentially performed until the last-level optimized query operation expression and last-level optimized query operation operator are obtained, and finally, the optimized query operation instruction is obtained according to the last-level optimized query operation expression and last-level optimized query operation operator.

[0097] The hierarchical optimization refers to hierarchical simplification or narrowing of the query data range according to different attribute information of the query operable compressed data. For example, in the primary optimization, the primary simplification or narrowing of the query data range can be performed according to the first attribute information of the query operable compressed data; in the secondary optimization, the secondary simplification or narrowing of the query data range can be performed according to the second attribute information of the query operable compressed data; and so on until the last optimization. It can be seen that the number of the hierarchical optimization is related to the number of the attribute information of the query operable compressed data. For example, if the original query operation expression is a<950, the optimization, i.e. narrowing of the query data range, can be performed according to the minimum value 900 of the statistical information of the query operable compressed data for the original query operation expression, and the obtained logical expression after the optimization can be a<950-900. For another example, if the original query operation expression is a<950 / A&a=1000 / B, i.e. the row group data less than 950 in the database A is queried and the row group data equal to 1000 in the database B is queried, the primary optimization, i.e. narrowing of the query data range, can be performed according to the minimum value 900 of the statistical information of the query operable compressed data of the database A for the original query operation expression, and the obtained primary optimization logical expression after the primary optimization can be a<950-900 / A&a=1000 / B, and then the secondary optimization, i.e. narrowing of the query data range, can be performed according to the greatest common divisor 100 of the statistical information of the query operable compressed data of the database B for the primary optimization logical expression, and the obtained secondary optimization logical expression after the secondary optimization can be a<950-900 / A&a=1000 / 100 / B.

[0098] Figure 7 FIG. 1 is a schematic diagram of a query operation instruction optimization flow according to an embodiment of the present application, Figure 7In the embodiment, assuming that the first attribute information is statistical information and the second attribute information is compression information, first, the query operation expression X0 and the query operation operator S0 corresponding to the query operation instruction are determined according to the query operation instruction, then the first-level optimization is performed on the query operation instruction according to the first attribute information, i.e. the statistical information, of the query operable compression data, to obtain the first-level optimized query operation expression X1 and the first-level optimized query operation operator S1, then the second-level optimization is performed on the first-level optimized query operation expression X1 according to the second attribute information, i.e. the compression information, of the query operable compression data, to obtain the second-level optimized query operation expression X2 and the second-level optimized query operation operator S2, in which the second-level optimized query operation expression X2 and the second-level optimized query operation operator S2 are the final-level optimized query operation expression and the final-level optimized query operation operator, and finally the optimized query operation instruction can be obtained according to the final-level optimized query operation expression X2 and the final-level optimized query operation operator S2. Of course, in actual operation, the attribute information for the hierarchical optimization can also be statistical information, but the statistical information is different.

[0099] In the embodiment of the present application, the method further comprises the step of performing operation on the query operable compression data according to the optimized query operation instruction, i.e. as shown in Figure 8 The method comprises the following steps S801-S804.

[0100] In step S801, the query operation instruction is acquired, and the corresponding query operation object data is acquired according to the query operation instruction, wherein the query operation object data is compression data carrying corresponding attribute information.

[0101] In step S802, the query operable compression data corresponding to the query operation object data is determined according to the query operation object data, and the attribute information of the query operable compression data is acquired.

[0102] In step S803, the query operation instruction is hierarchically optimized according to the attribute information of the query operable compression data, to obtain the optimized query operation instruction.

[0103] In step S804, operation is performed on the query operable compression data according to the optimized query operation instruction.

[0104] After the query operation instruction is optimized, the optimized query operation instruction can be directly used to perform operation on the query operable compression data, and the operation based on the compression data can effectively avoid the data decompression process, save the storage space occupied by the decompressed data, simplify the data operation process, and improve the data operation performance.

[0105] In the embodiment of the present application, asFigure 9 As shown, the step S804, i.e. the step of performing operation on the query operable compressed data according to the optimized query operation instruction, comprises steps S901-S902:

[0106] In step S901, target query operable compressed data in the query operable compressed data is determined according to the optimized query operation instruction.

[0107] In step S902, operation is performed on the target query operable compressed data according to the optimized query operation instruction.

[0108] After the query operation instruction is optimized, the corresponding operation data object will change, for example, the storage range of the data object may be reduced, at this time, the target query operable compressed data corresponding to the optimized query operation instruction needs to be determined again, and then operation is performed on the target query operable compressed data according to the optimized query operation instruction to obtain the operation result.

[0109] Next, the technical solutions provided by the embodiments of the present application are explained and described by taking several examples.

[0110] Example 1: the query operation instruction is to query row group data less than 950 in database A, the corresponding logical expression can be expressed as a<950, the query operation object data corresponding to the query operation instruction is the numerical row group data stored in database A, and the logical table is shown as Figure 10A As shown, a lot of numerical data is stored in database A, for example, Figure 10B As shown, the original data to be stored is 64-bit integer data type (int64) data with a minimum value of 900, i.e. 910, 900, 999, etc., when the FOR compression method is used for compression storage, the original data 910, 900, 999, etc. can be compressed into operable 8-bit integer data type data 10, 0, 99, etc. obtained by subtracting the minimum value 900, and the minimum value 900 is stored as statistical information of the compressed data. In the prior art, when querying row group data less than 950 in database A, the 8-bit integer data type data stored in a certain row group in database A needs to be converted into 64-bit integer data type data, then the minimum value 900 is added respectively to obtain 64-bit integer data type data, and then compared with 950, for example, Figure 10CThe technical scheme of the embodiment of the application does not need the above data decompression process, but only needs to optimize the logical expression a<950 and the corresponding operator a<950 according to the statistical information of the compressed data, i.e. the minimum value 900, to obtain the optimized logical expression a<950-900 and the corresponding operator a<50, and then the operation of a<950 can be completed by directly querying the 8-bit integer data type data less than 50 in the row group of the database A, at this time, the target query can operate the compressed data which is the 8-bit integer data type data less than 50, and the optimized logical table is shown in Figure 10D In the example, the constant 950 in the expression a<950 is transferred to 950-minimum value, and the constant is combined by substituting the minimum value 900, i.e. 950-900=50, and since the optimized logical expression can be directly based on the 8-bit integer data type compressed data, compared with the prior art which needs to restore the 8-bit integer data type compressed data to the 64-bit integer data type original data to perform the operation, the dimension reduction of the operator is realized, and the data operation after the dimension reduction not only greatly saves the data storage space, but also greatly saves the data operation time.

[0111] In example 2, the query operation instruction is to query the row group data equal to 1000 in the database B, and the corresponding logical expression can be expressed as a=1000, and the query operation object data corresponding to the query operation instruction is the numerical row group data stored in the database B. The database B stores a lot of numerical data, as shown in Figure 11A The original data to be stored is 800, 900, 1000, etc. 64-bit integer data type data with a common divisor 100, and when the GCD compression method is used for compression storage, the original data 800, 900, 1000, etc. can be compressed into the 8-bit integer data type data 8, 9, 10, etc. which can be operated by removing the common divisor 100, and the common divisor 100 is stored as the statistical information of the row group compressed data. In the prior art, when the row group data equal to 1000 in the database B is queried, the 8-bit integer data type data stored in the row group of the database B needs to be converted into 64-bit integer data type data, then multiplied by the common divisor 100 to obtain the 64-bit integer data type data, and then compared with 1000, as shown in Figure 11Bshown. With the technical solution of the embodiment of the application, without the data decompression process, only according to the statistical information of the compressed data, i.e. the common divisor 100, the logical expression a=1000 and the corresponding operator a=1000 are optimized to obtain the optimized logical expression a=1000 / 100 and the corresponding operator a=10, and then the operation of a=1000 can be completed by directly querying the 8-bit integer data type data equal to 10 in the row group of the database B, at this time, the target query can be operated on the compressed data equal to 10 of the 8-bit integer data type, and the optimized logical table is shown in Figure 11C In this example, the constant 1000 in the expression a=1000 is transferred to 1000 / common divisor, and the merging of the constant is realized by substituting the common divisor 100: 1000 / 100=10, and since the optimized logical expression can be directly based on the 8-bit integer data type compressed data, compared with the prior art in which the 8-bit integer data type compressed data needs to be restored to the original 64-bit integer data type data for operation, the dimension reduction of the operator is realized, and the data operation after dimension reduction not only greatly saves the data storage space, but also greatly saves the data operation time. For example, for the database B, if the query operation instruction is to query the row group data less than or equal to 1001 in the database B, in the prior art, the 8-bit integer data type data stored in the database B needs to be converted to 64-bit integer data type data, then multiplied by the common divisor 100 to obtain 64-bit integer data type data, and then compared with 1001, as shown in Figure 11D With the technical solution of the embodiment of the application, without the data decompression process, only according to the statistical information of the compressed data, i.e. the common divisor 100, the logical expression a=1000 and the corresponding operator a=1000 are optimized to obtain the optimized logical expression a=1000 / 100 and the corresponding operator a=10, and then the operation of a=1000 can be completed by directly querying the 8-bit integer data type data equal to 10 in the row group of the database B, at this time, the target query can be operated on the compressed data equal to 10 of the 8-bit integer data type, and the optimized logical table is shown in Figure 11E

[0112] Example 3, the query operation instruction is to calculate the sum of two row group data in the database C, and the corresponding logical expression can be expressed as a+b, as shown in Figure 12A shown, the row group of a uses the FOR compression method, and the minimum value is 900, the original data 910, 900, 999…64-bit integer data type data, and the operable 8-bit integer data type data 10, 0, 99… is obtained by using the FOR compression method for storage, and the minimum value 900 is the statistical information of the row group. As shown in Figure 12B ​As shown, the row group where b is located also uses the FOR compression method, and the minimum value is 2000. The original data 2000, 3000, 2500, … is of the 64-bit integer data type, and after compression using the FOR compression method, the operable 16-bit integer data type data 0, 1000, 500, … is obtained and stored. The minimum value 2000 is the statistical information of the row group. In the prior art, when the sum of the data of two row groups in the database C is calculated, the 8-bit integer data type data stored in a row group in the database C needs to be converted into 64-bit integer data type data, and then the minimum value 900 is added to obtain 64-bit integer data type data. The 16-bit integer data type data stored in another row group in the database C needs to be converted into 64-bit integer data type data, and then the minimum value 2000 is added to obtain 64-bit integer data type data, as shown in the following formula: Figure 12C According to the technical solution of the embodiment of the application, the above data decompression process is not needed. Only the statistical information of the compressed data, that is, the minimum values 900 and 2000, is needed to optimize the logical expression a+b and the corresponding operator a+b to obtain the optimized logical expression a+b+minimum value 1+minimum value 2 and the corresponding operator a+b+2900. However, considering that the data storage type of the row group where a is located is 8-bit integer data, and the data storage type of the row group where b is located is 16-bit integer data, the data of the row group where a is located still needs to be converted into 16-bit integer data that can be calculated with the data of the row group where b is located, and then the sum of the minimum values 2900 is added to obtain the operation result of a+b. The logical table of the optimized expression is shown in the following figure: Figure 12D If the finally calculated sum exceeds the accommodation range of the 16-bit integer data type, the sum can be converted into 64-bit integer data type, as shown in the following figure: Figure 12D The dashed box shows. This example effectively realizes the dimension reduction of the operator, and the data operation after the dimension reduction greatly saves the data storage space and greatly saves the data operation time.

[0113] Example 4, the query operation instruction is to calculate the sum of the data of a row group in the database D. The corresponding logical expression can be expressed as SUM(a), as shown in the following figure: Figure 13A As shown, the row group where a is located uses the FOR compression method, and the minimum value is 900. The original data 910, 900, 999, … is of the 64-bit integer data type, and after compression using the FOR compression method, the operable 8-bit integer data type data 10, 0, 99, … is obtained and stored. The minimum value 900 is the statistical information of the row group. In the prior art, when the sum of the data of a row group in the database D is calculated, the 8-bit integer data type data stored in the row group in the database D needs to be converted into 64-bit integer data type data, and then the minimum value 900 is added to obtain 64-bit integer data type data, and then the sum is calculated, as shown in the following formula: Figure 13BAs shown in the figure. Using the technical solution of this invention, the above-mentioned data decompression process is unnecessary. It only requires optimizing the logical expression SUM(a) and the corresponding operator SUM(a) based on the statistical information of the compressed data, i.e., the minimum value 900, to obtain the optimized logical expression SUM(a) + minimum value × N, where N is the number of data in the row group, and the corresponding operator SUM(a) + 900 × N. Considering that 8-bit integer data types may need to be converted after summation, and that the minimum value × N belongs to 64-bit integer data types, during the optimization operation, the 8-bit integer data types can be converted to a data type lower than 64 bits first, summed, and then the resulting sum converted to 64-bit integer data types. This sum is then combined with the minimum value × N, which also belongs to 64-bit integer data types. The optimized logical expression is illustrated in the figure. Figure 13C As shown in the example, this example implements dimensionality reduction of operators during the computation process, and performs corresponding operations in advance in the low-dimensional case, which greatly reduces both data storage space and data processing time.

[0114] Example 5: The query operation command is to calculate the sum of a certain row of data in database E. The corresponding logical expression can be represented as SUM(a), where, for example... Figure 14A As shown, the row group containing 'a' uses GCD compression, and its common divisor is 100. The original data is 64-bit integer data of 800, 900, 1000... After compression using GCD, it is stored as operable 8-bit integer data of 8, 9, 10... The common divisor 200 represents the statistical information of this row group. In the prior art, when calculating the sum of a row group in database E, it is necessary to convert the 8-bit integer data of that row group stored in database E into 64-bit integer data, then multiply each data by the common divisor 100, and then perform the summation calculation, as follows... Figure 14B As shown in the figure. Using the technical solution of this invention, the above-mentioned data decompression process is unnecessary. It only requires optimizing the logical expression SUM(a) and the corresponding operator SUM(a) based on the statistical information of the compressed data, i.e., the common divisor 100, to obtain the optimized logical expression SUM(a) × common divisor and the corresponding operator SUM(a) × common divisor. Considering that 8-bit integer data types may need to be converted after summation, during the optimization operation, the 8-bit integer data type can be converted to a data type lower than 64 bits first, summed, and then the resulting sum converted to a 64-bit integer data type and multiplied by the common divisor. The optimized logical expression is illustrated in the figure. Figure 14CThe example realizes dimension reduction of the operator in the calculation process, and performs the corresponding operation operation in advance in the low-dimensional case, so that the data storage space and the data operation time are greatly reduced.

[0115] The following is an embodiment of the device of the application, which can be used to execute the method embodiment of the application.

[0116] Figure 15 The structure block diagram of the query operation instruction optimization device according to an embodiment of the application is shown, which can be realized as part or all of an electronic device by software, hardware or combination of the two, and can be realized as a query operation instruction optimization server. Figure 15 As shown in the figure, the query operation instruction optimization device comprises:

[0117] The acquisition module 1501 is configured to acquire the query operation instruction and acquire the corresponding query operation object data according to the query operation instruction, wherein the query operation object data is compressed data carrying corresponding attribute information;

[0118] The determination module 1502 is configured to determine the query operable compressed data corresponding to the query operation object data according to the query operation object data, and acquire the attribute information of the query operable compressed data;

[0119] The optimization module 1503 is configured to perform hierarchical optimization on the query operation instruction according to the attribute information of the query operable compressed data, and obtain the optimized query operation instruction.

[0120] As mentioned above, with the development of data technology, more and more data needs to be stored in the database. In order to save the data storage space, it is usually necessary to compress the data to be stored before storage, but when subsequent data query and other data operations are performed, the compressed data cannot be directly operated, and the query operation needs to be performed after the compressed data is completely decompressed and restored to the original data. The tedious data decompression process greatly reduces the query performance of the engine. The solution in the prior art either does not optimize the execution process of data operation and the occupation of memory in essence, or has very limited application occasions.

[0121] In view of the above problems, in this embodiment, a query operation instruction optimization device is proposed, which performs hierarchical optimization on the query operation instruction by means of the attribute information of the query operation object data, to obtain the optimized query operation instruction, so as to directly operate the compressed data by using the optimized query operation instruction subsequently. The technical scheme has strong applicability, avoids the tedious data decompression process, greatly saves the storage space of the decompressed data, simplifies the data operation process, and improves the data operation performance.

[0122] In an embodiment of the present application, the query operation instruction refers to a query instruction for realizing a certain operation purpose, such as a query instruction, a search instruction, etc. In the field of SQL, the query operation instruction can be realized as a SQL query operation instruction.

[0123] In an embodiment of the present application, the query operation object data corresponding to the query operation instruction refers to data related to the query operation instruction and possibly eventually serving as the object of the query operation. For example, if the query operation instruction is to query a number less than 950 in a database, all the data stored in the database can be considered as the query operation object data corresponding to the query operation instruction, which can possibly participate in the operation of the query operation in the subsequent query operation.

[0124] In an embodiment of the present application, since the query operation object data is data already stored in the database, it is compressed data, and the compressed data also carries corresponding attribute information, wherein the attribute information can include one or more of the following information: statistical information of the compressed data, compression information of the compressed data, storage information of the compressed data, etc. The presence of the attribute information makes it possible for the present application to optimize the query operation instruction, wherein the statistical information can be, for example: the maximum value of the data, the minimum value of the data, the sum value of the data within a certain preset range, the data type, etc. The compression information can be, for example: the original data type, the data storage type, the data compression method, etc. The storage information can be, for example: the storage type, the storage location, etc.

[0125] In an embodiment of the present application, the compressed data can be, for example, compressed data stored column by column in units of row groups in a HiStore columnar database, wherein a row group refers to a data unit formed by a plurality of rows in columnar storage, and here the row refers to a data row in the traditional sense in the field of databases, that is, in the HiStore columnar database, data is stored column by column, and in a certain column, the data is further divided into row groups composed of a plurality of rows. In addition, the data in the row group can be independently compressed and information- counted in units of row groups, wherein the result of information counting is the statistical information or attribute information of the row group, which is mainly used to describe the characteristics and attributes of the stored data of the row group. Therefore, in this embodiment, the compressed data includes a compressed data packet for storing the compressed data and a compressed data information packet for storing the attribute information of the compressed data. Figure 2 A schematic diagram of column-by-column storage of compressed data according to an embodiment of the present application is shown in Figure 2 The data storage table includes column 1, column 2, column 3, and other columns (not shown in Figure 2 The data storage table includes row group 1, row group 2, row group 3, and other row groups (not shown in Figure 2 In this embodiment, the compressed data in each column is stored in units of row groups, and the compressed data in each row group is stored in units of rows.Figure 2 In the figure, the intersection of a column and a row group represents a row group unit contained in the column, and the black box in each intersection represents the statistical information of the row group, i.e., a compressed data information package, and the gray box represents a data package formed by compressing each row data in the row group, i.e., a compressed data package.

[0126] In an embodiment of the present application, the operable compressed data refers to data that can be directly queried and the like on the basis of compressed data, and can also be referred to as interpretable compressed data. For example, compressed data obtained by using a compression method such as Delta compression based on an increment, GCD compression based on a greatest common divisor, FOR (Frame-of-reference) compression, and various Dictionary compression methods are operable compressed data. Similarly, the opposite of the operable compressed data is inoperable compressed data, which refers to data that cannot be directly queried and the like on the basis of compressed data, and needs to be decompressed into operable compressed data before being queried and the like, and can also be referred to as ininterpretable compressed data. For example, compressed data obtained by using a compression method such as Huffman coding, arithmetic coding, and LZ77 / LZ4 algorithm are inoperable compressed data. Figure 3 A flowchart of data compression and compressed data query in the prior art is shown in FIG. 1. Figure 3 As shown in FIG. 1, in general, when original data is compressed into compressed data and stored in a disk, the original data can be first compressed into operable compressed data with a certain compression rate, and then the operable compressed data is compressed into inoperable compressed data with a higher compression rate for storage. When it is necessary to query the compressed data, the inoperable compressed data needs to be decompressed into operable compressed data with a certain readability, and then the operable compressed data is decompressed and restored into the original data for data query operation.

[0127] In an embodiment of the present application, the query operation instruction is hierarchically optimized according to the attribute information of the query operable compressed data, which means that the query operation instruction is simplified and optimized in different aspects according to the obtained attribute information of the query operable compressed data, so as to finally obtain an optimized query operation instruction. The specific optimization process of the query operation instruction will be described in detail below.

[0128] In an embodiment of the present application, as shown in FIG. 15, the determining module 1502 includes: Figure 16

[0129] The first determining submodule 1601 is configured to determine the query operation object data as the query operable compressed data corresponding thereto when the query operation object data is operable compressed data.

[0130] ​The second determining submodule 1602 is configured to, when the query operation object data is the inoperable compressed data, decompress the inoperable compressed data until the operable compressed data is obtained, and determine the decompressed operable compressed data as the query operable compressed data corresponding to the query operation object data.

[0131] The obtaining submodule 1603 is configured to obtain attribute information of the query operable compressed data, wherein the attribute information of the query operable compressed data at least includes one or more of the following information: statistical information, compression information, and storage information.

[0132] As mentioned above, in the prior art, when the stored compressed data is queried, the inoperable compressed data needs to be decompressed into the operable compressed data with a certain readability, and then the operable compressed data needs to be decompressed and restored into the original data for the data query operation. In order to avoid the tedious data decompression process, save the storage space of the decompressed data, simplify the data operation process, and improve the data operation performance, in the embodiment, in order to be able to directly perform the corresponding operation on the compressed data in the future, it is necessary to first determine whether the query operation object data is the operable compressed data. If the previously obtained query operation object data is already the operable compressed data, the query operation object data is determined as the query operable compressed data corresponding thereto, and the subsequent data operation such as query can be directly performed. If the query operation object data is the inoperable compressed data, the inoperable compressed data needs to be decompressed until the operable compressed data corresponding thereto is obtained.

[0133] Figure 5 As shown in the flowchart of the data compression and compressed data query according to the embodiment of the present application, Figure 5 When the compressed data is queried, only the inoperable compressed data stored in the database needs to be decompressed into the operable compressed data with a certain readability, and the operable compressed data does not need to be decompressed and restored into the original data, so that the data query operation can be performed.

[0134] In the embodiment of the present application, as shown in Figure 17 The optimization module 1503 includes:

[0135] The third determining submodule 1701 is configured to determine the query operation expression and the query operation operator corresponding to the query operation instruction according to the query operation instruction;

[0136] The first optimization submodule 1702 is configured to obtain the first attribute information of the query operable compressed data, perform the first-level optimization on the query operation instruction according to the first attribute information, and obtain the first-level optimized query operation expression and the first-level optimized query operation operator corresponding thereto.

[0137] The second optimization submodule 1703 is configured to acquire second attribute information of the query operable compressed data, perform secondary optimization on the first-level optimized query operation expression according to the second attribute information, obtain corresponding secondary optimized query operation expression and secondary optimized query operation operator, and perform optimization on the query operation instruction according to the last-level optimized query operation expression and last-level optimized query operation operator until the last-level optimized query operation expression and last-level optimized query operation operator are obtained.

[0138] In this embodiment, when performing hierarchical optimization on the query operation instruction according to attribute information of the query operable compressed data, first, the query operation expression and query operation operator corresponding to the query operation instruction are determined according to the query operation instruction; then, first attribute information of the query operable compressed data is acquired, the query operation instruction is optimized according to the first attribute information, and corresponding first-level optimized query operation expression and first-level optimized query operation operator are obtained; then, second attribute information of the query operable compressed data is acquired, the first-level optimized query operation expression is optimized according to the second attribute information, and corresponding secondary optimized query operation expression and secondary optimized query operation operator are obtained; and the process is sequentially performed until the last-level optimized query operation expression and last-level optimized query operation operator are obtained, and finally, the optimized query operation instruction is obtained according to the last-level optimized query operation expression and last-level optimized query operation operator.

[0139] The hierarchical optimization refers to hierarchical simplification or narrowing of the query data range according to different attribute information of the query operable compressed data. For example, in the primary optimization, the primary simplification or narrowing of the query data range can be performed according to the first attribute information of the query operable compressed data; in the secondary optimization, the secondary simplification or narrowing of the query data range can be performed according to the second attribute information of the query operable compressed data; and so on until the last optimization. It can be seen that the number of the hierarchical optimization is related to the number of the attribute information of the query operable compressed data. For example, if the original query operation expression is a<950, the optimization, i.e. narrowing of the query data range, can be performed according to the minimum value 900 of the statistical information of the query operable compressed data for the original query operation expression, and the obtained logical expression after the optimization can be a<950-900. For another example, if the original query operation expression is a<950 / A&a=1000 / B, i.e. the row group data less than 950 in the database A is queried and the row group data equal to 1000 in the database B is queried, the primary optimization, i.e. narrowing of the query data range, can be performed according to the minimum value 900 of the statistical information of the query operable compressed data of the database A for the original query operation expression, and the obtained primary optimization logical expression after the primary optimization can be a<950-900 / A&a=1000 / B, and then the secondary optimization, i.e. narrowing of the query data range, can be performed according to the greatest common divisor 100 of the statistical information of the query operable compressed data of the database B for the primary optimization logical expression, and the obtained secondary optimization logical expression after the secondary optimization can be a<950-900 / A&a=1000 / 100 / B.

[0140] Figure 7 FIG. 1 is a schematic diagram of a query operation instruction optimization flow according to an embodiment of the present application, Figure 7In the embodiment, assuming that the first attribute information is statistical information and the second attribute information is compression information, first, the query operation expression X0 and the query operation operator S0 corresponding to the query operation instruction are determined according to the query operation instruction, then the first-level optimization is performed on the query operation instruction according to the first attribute information, i.e., the statistical information, of the query operable compression data, to obtain the first-level optimized query operation expression X1 and the first-level optimized query operation operator S1, and then the second-level optimization is performed on the first-level optimized query operation expression X1 according to the second attribute information, i.e., the compression information, of the query operable compression data, to obtain the second-level optimized query operation expression X2 and the second-level optimized query operation operator S2, in which the second-level optimized query operation expression X2 and the second-level optimized query operation operator S2 are the final-level optimized query operation expression and the final-level optimized query operation operator, and finally the optimized query operation instruction can be obtained according to the final-level optimized query operation expression X2 and the final-level optimized query operation operator S2. Of course, in actual operation, the attribute information for the hierarchical optimization can also be statistical information, but the statistical information is different.

[0141] In the embodiment of the application, the device further comprises a part for executing operation on the query operable compression data according to the optimized query operation instruction, i.e., as shown in Figure 18 The device comprises:

[0142] The acquisition module 1801 is configured to acquire a query operation instruction and acquire corresponding query operation object data according to the query operation instruction, wherein the query operation object data is compression data carrying corresponding attribute information.

[0143] The determination module 1802 is configured to determine query operable compression data corresponding to the query operation object data according to the query operation object data and acquire attribute information of the query operable compression data.

[0144] The optimization module 1803 is configured to perform hierarchical optimization on the query operation instruction according to the attribute information of the query operable compression data to obtain an optimized query operation instruction.

[0145] The execution module 1804 is configured to execute operation on the query operable compression data according to the optimized query operation instruction.

[0146] After the query operation instruction is optimized, the optimized query operation instruction can be used to directly execute operation on the query operable compression data, and the operation based on the compression data can effectively avoid data decompression process, save storage space occupied by decompressed data, simplify data operation process and improve data operation performance.

[0147] In an embodiment of the present application, as shown in Figure 19 The execution module 1804 includes:

[0148] The fourth determination sub-module 1901 is configured to determine target query operable compressed data in the query operable compressed data according to the optimized query operation instruction.

[0149] The execution sub-module 1902 is configured to execute operation on the target query operable compressed data according to the optimized query operation instruction.

[0150] After the optimization of the query operation instruction, the corresponding operation data object will change, for example, the storage range of the data object may be reduced, at this time, the target query operable compressed data corresponding to the optimized query operation instruction needs to be determined again, and then operation is executed on the target query operable compressed data according to the optimized query operation instruction to obtain the operation result.

[0151] The embodiment of the present application further discloses an electronic device, Figure 20 A structural block diagram of an electronic device according to an embodiment of the present application is shown, as shown in Figure 20 The electronic device 2000 includes a memory 2001 and a processor 2002; wherein,

[0152] The memory 2001 is used for storing one or more computer instructions, wherein the one or more computer instructions are executed by the processor 2002 to realize any method step described above.

[0153] Figure 21 A structural schematic diagram of a computer system suitable for realizing the query operation instruction optimization method according to the embodiment of the present application.

[0154] As shown in Figure 21 The computer system 2100 includes a processing unit 2101, which can execute various processes in the above embodiments according to programs stored in a read-only memory (ROM) 2102 or loaded into a random access memory (RAM) 2103 from a storage part 2108. Various programs and data required for the operation of the system 2100 are also stored in the RAM 2103. The processing unit 2101, the ROM 2102 and the RAM 2103 are connected to each other through a bus 2104. An input / output (I / O) interface 2105 is also connected to the bus 2104.

[0155] The following components are connected to the I / O interface 2105: an input part 2106 including a keyboard, a mouse, etc.; an output part 2107 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 2108 including a hard disk, etc.; and a communication part 2109 including a network interface card such as a LAN card, a modem, etc. The communication part 2109 performs communication processing via a network such as the Internet. A drive 2110 is also connected to the I / O interface 2105 as necessary. A removable media 2111 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 2110 as necessary, so that a computer program read out therefrom is installed in the storage part 2108 as necessary.

[0156] In particular, the method described above can be implemented as a computer software program according to embodiments of the present application. For example, embodiments of the present application include a computer program product comprising a computer program tangibly embodied on a non-transitory computer readable medium, the computer program containing program code for executing the query operation instruction optimization method. In such embodiments, the computer program can be downloaded and installed from a network via the communication part 2109, and / or installed from the removable media 2111. The processing unit 1501 can be implemented as a CPU, GPU, FPGA, NPU, etc. processing unit.

[0157] The flow and block diagrams in the drawings show possible architectures, functional and operational, of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flow and block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or combinations of hardware and software.

[0158] The units or modules described in the embodiments of the present application can be implemented by means of software, or by means of hardware. The described units or modules can also be provided in a processor, and the names of these units or modules do not constitute a limitation on the units or modules themselves in some cases.

[0159] As another aspect, the embodiments of the present application also provide a computer readable storage medium, which can be the computer readable storage medium included in the apparatus in the above-mentioned embodiments, or can exist separately and not be assembled into the apparatus. The computer readable storage medium stores one or more programs for being executed by one or more processors to perform the method described in the embodiments of the present application.

[0160] The above description is merely the preferred embodiments of the present application and the explanation of the applied technical principles. It should be understood by those skilled in the art that the inventive scope of the embodiments of the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by the combinations of the above technical features or their equivalent features without departing from the inventive concept. For example, the technical solutions formed by the mutual replacement of the above features and the technical features with similar functions disclosed in the embodiments of the present application (but not limited to) can also be covered.​

Claims

1. A method for optimizing query operation instructions, characterized in that, The method comprises the following steps: acquiring a query operation instruction and acquiring corresponding query operation object data according to the query operation instruction, wherein the query operation object data is compressed data carrying corresponding attribute information; determining query operable compressed data corresponding to the query operation object data according to the query operation object data and acquiring attribute information of the query operable compressed data; performing hierarchical optimization on the query operation instruction according to the attribute information of the query operable compressed data to obtain an optimized query operation instruction; the hierarchical optimization refers to hierarchical simplification or reduction of a query operation expression according to different attribute information of the query operable compressed data; the query operation expression is an expression corresponding to the query operation instruction.

2. The method of claim 1, wherein, The compressed data is stored column by column in units of row groups, and the compressed data comprises compressed data packets for storing compressed data and compressed data information packets for storing attribute information of the compressed data.

3. The method according to claim 1 or 2, characterized in that, The step of determining query operable compressed data corresponding to the query operation object data according to the query operation object data and acquiring attribute information of the query operable compressed data comprises the following steps: when the query operation object data is operable compressed data, determining the query operation object data as the query operable compressed data corresponding thereto; when the query operation object data is inoperable compressed data, decompressing the inoperable compressed data until operable compressed data is obtained, and determining the obtained operable compressed data as the query operable compressed data corresponding to the query operation object data; acquiring attribute information of the query operable compressed data, wherein the attribute information of the query operable compressed data comprises one or more of the following information: statistical information, compression information and storage information.

4. The method of claim 3, wherein, The step of performing hierarchical optimization on the query operation instruction according to the attribute information of the query operable compressed data to obtain an optimized query operation instruction comprises the following steps: determining a query operation expression and a query operation operator corresponding to the query operation instruction according to the query operation instruction; acquiring first attribute information of the query operable compressed data, performing first-level optimization on the query operation instruction according to the first attribute information to obtain a first-level optimized query operation expression and a first-level optimized query operation operator; acquiring second attribute information of the query operable compressed data, performing second-level optimization on the first-level optimized query operation expression according to the second attribute information to obtain a second-level optimized query operation expression and a second-level optimized query operation operator, and repeating the above steps until a final-level optimized query operation expression and a final-level optimized query operation operator are obtained, and obtaining the optimized query operation instruction according to the final-level optimized query operation expression and the final-level optimized query operation operator.

5. The method of claim 4, wherein, The method further comprises the following steps: performing operation on the query operable compressed data according to the optimized query operation instruction.

6. The method of claim 5, wherein, The step of performing operation on the query operable compressed data according to the optimized query operation instruction comprises the following steps: determining target query operable compressed data in the query operable compressed data according to the optimized query operation instruction; According to the optimized query operation instruction, operation is performed on the target query operable compressed data.

7. A query operation instruction optimization apparatus characterized by comprising: Comprise: An acquisition module configured to acquire a query operation instruction and acquire corresponding query operation object data according to the query operation instruction, wherein the query operation object data is compressed data carrying corresponding attribute information; A determination module configured to determine query operable compressed data corresponding to the query operation object data according to the query operation object data and acquire attribute information of the query operable compressed data; An optimization module configured to perform hierarchical optimization on the query operation instruction according to the attribute information of the query operable compressed data to obtain an optimized query operation instruction; the hierarchical optimization refers to hierarchical simplification or narrowing of query data range according to query operation expressions according to different attribute information of the query operable compressed data; the query operation expression is an expression corresponding to the query operation instruction.

8. The apparatus of claim 7, wherein, The compressed data is stored column by column in units of row groups, and the compressed data includes compressed data packets for storing compressed data and compressed data information packets for storing compressed data attribute information.

9. The apparatus of claim 7 or 8, wherein, The determination module comprises: A first determination submodule configured to determine the query operation object data as query operable compressed data corresponding thereto when the query operation object data is operable compressed data; A second determination submodule configured to decompress the inoperable compressed data until operable compressed data is obtained when the query operation object data is inoperable compressed data, and determine the decompressed operable compressed data as query operable compressed data corresponding to the query operation object data; An acquisition submodule configured to acquire attribute information of the query operable compressed data, wherein the attribute information of the query operable compressed data at least includes one or more of the following information: statistical information, compression information, and storage information.

10. The apparatus of claim 9, wherein, The optimization module comprises: A third determination submodule configured to determine a query operation expression and a query operation operator corresponding to the query operation instruction according to the query operation instruction; A first optimization submodule configured to acquire first attribute information of the query operable compressed data, perform first-level optimization on the query operation instruction according to the first attribute information to obtain a corresponding first-level optimized query operation expression and a first-level optimized query operation operator; A second optimization submodule configured to acquire second attribute information of the query operable compressed data, perform second-level optimization on the first-level optimized query operation expression according to the second attribute information to obtain a corresponding second-level optimized query operation expression and a second-level optimized query operation operator, and perform hierarchical optimization until a final-level optimized query operation expression and a final-level optimized query operation operator are obtained, and obtain the optimized query operation instruction according to the final-level optimized query operation expression and the final-level optimized query operation operator.

11. The apparatus of claim 10, wherein, Further comprising: An execution module configured to perform operation on the query operable compressed data according to the optimized query operation instruction.

12. The apparatus of claim 11, wherein, The execution module comprises: A fourth determining sub-module, configured to determine a target query operable compressed data in the query operable compressed data according to the optimized query operation instruction; An executing sub-module, configured to execute operation on the target query operable compressed data according to the optimized query operation instruction.

13. An electronic device, comprising: comprising a memory and a processor; wherein, The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method steps of any one of claims 1-6.

14. A computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions, when executed by the processor, implement the method steps of any one of claims 1-6.

Images