Database operation statement processing method, medium, product and equipment

Abstract

The invention provides a database operation statement processing method, a medium, a product and equipment. The method comprises the following steps: acquiring a database operation statement; judging whether the database operation statement contains a duplicate removal keyword or not; if yes, whether the database operation statement meets a preset optimization condition or not is judged; under the condition that a preset optimization condition is not met, a serial path and a parallel path are created for the database operation statement, the parallel path is used for configuring a plurality of parallel threads and a summary thread, and the parallel thread is used for processing other operations except the duplicate removal operation in the database operation statement in parallel; judging whether the parallel cost required for executing the parallel path is smaller than the serial cost required for executing the serial path or not; and if yes, selecting the parallel path as an execution path of the database operation statement. According to the method, under the condition that duplicate removal keywords cannot be eliminated, a serial path and a parallel path can be created, and one path with low cost is selected for execution, so that the duplicate removal efficiency of the database is improved.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a method, medium, product and device for processing database operation statements. Background Art

[0002] In current database technology, duplicate values ​​may appear in the query results after using database operation statements to query. When non-duplicate results are required, the deduplication keyword distinct is generally used to return unique and different results. When the optimizer processes database operation statements with deduplication keywords, it will perform deduplication operations on the results regardless of whether there are duplicate tuples.

[0003] In order to improve the efficiency of deduplication by using parallelism, databases usually use logical conversion to convert distinct operations into group by operations for parallel operation. However, in some special cases, distinct operations cannot be converted into group by operations. Therefore, a method is needed to solve the problem that the conversion of distinct to group by operations cannot be performed due to some constraints, resulting in the inability to implement parallel deduplication. Summary of the invention

[0004] In view of the above problems, the present invention proposes a method, medium, product and device for processing database operation statements that overcome the above problems or at least partially solve the above problems.

[0005] An object of the present invention is to improve the efficiency of deduplication of database operation statements.

[0006] A further object of the present invention is to improve the execution speed through parallel processing even when the distinct operation cannot be converted into a group by.

[0007] Another further object of the present invention is to select an execution path with the lowest cost to execute a database operation statement when duplicate keywords cannot be eliminated.

[0008] In particular, the present invention provides a method for processing a database operation statement, comprising: Get database operation statements; Determine whether the database operation statement contains deduplication keywords; If yes, then determine whether the database operation statement meets the preset optimization conditions, where the preset optimization conditions are the conditions required to eliminate duplicate keywords; When the database operation statement does not meet the preset optimization conditions, a serial path and a parallel path are created for the database operation statement respectively, wherein the parallel path is used to configure multiple parallel threads and a summary thread, the parallel threads are used to parallelly process other operations in the database operation statement except the deduplication operation, and the summary thread is used to perform the deduplication operation after summarizing the execution results of the parallel threads; Determine whether the parallel cost required to execute the parallel path is less than the serial cost required to execute the serial path; If so, select the parallel path as the execution path of the database operation statement.

[0009] Optionally, the step of creating a parallel path for a database operation statement includes: Determine the degree of parallelism corresponding to the parallel path; The same number of parallel threads and a summary thread are created according to the parallelism degree.

[0010] Optionally, the step of determining whether the parallel cost required to execute the parallel path is less than the serial cost required to execute the serial path includes: estimating the serial cost required to execute the serial path and the parallel cost required to execute the parallel path; The steps of estimating the serial cost required to execute the serial path include: determining the type of operation to be performed, the data column to be operated, and the data table corresponding to the data column in the deduplication process according to the database operation statement; determining the amount of data to be executed and the complexity of the operation through the type of operation to be performed, the data column to be operated, and the data table corresponding to the data column; and calculating the serial cost based on the amount of data and the complexity of the operation.

[0011] Optionally, the step of estimating the parallel cost required to execute the parallel path includes: Determine the type of operation to be performed, the data column to be operated, and the data table corresponding to the data column in the deduplication process according to the database operation statement; Determine the amount of data to be executed and the complexity of the operation by the type of operation to be executed, the data column to be operated, and the data table corresponding to the data column; The parallel thread cost corresponding to the execution of the parallel thread and the summary thread cost corresponding to the execution of the summary thread are calculated based on the data volume, operation complexity and parallelism; Combine the parallel thread cost and the aggregate thread cost as the parallel cost.

[0012] Optionally, the step of determining whether the database operation statement meets the preset optimization condition includes: Determine the data column for deduplication keyword operation in the database operation statement and the data table corresponding to the data column; Determine column by column whether the data columns in the corresponding data table meet the non-empty conditions and unique conditions; When all data columns in the corresponding data table meet the non-empty condition and the uniqueness condition, it is determined that the database operation statement meets the preset optimization condition.

[0013] Optionally, the step of determining whether the database operation statement meets the preset optimization condition includes: When the database operation statement meets the preset optimization conditions, the duplicate keywords are eliminated and the optimized database operation statement is executed.

[0014] Optionally, before the step of determining whether the database operation statement contains a deduplication keyword, the step further includes: Determine whether the database operation statement is a query statement; If so, the step of determining whether the database operation statement contains duplicate removal keywords is executed.

[0015] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods for processing database operation statements are implemented.

[0016] According to another aspect of the present invention, a computer program product is provided, which includes a computer program, and when the computer program is executed by a processor, the steps of any one of the above-mentioned methods for processing database operation statements are implemented.

[0017] According to another aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a machine executable program stored in the memory and running on the processor, and the processor implements the steps of any of the above-mentioned database operation statement processing methods when executing the machine executable program.

[0018] The processing method of the database operation statement of the present invention first obtains the database operation statement; then determines whether the database operation statement contains deduplication keywords; if so, determines whether the database operation statement meets the preset optimization conditions, and the preset optimization conditions are the conditions required for eliminating the deduplication keywords; when the database operation statement does not meet the preset optimization conditions, creates a serial path and a parallel path for the database operation statement respectively, wherein the parallel path is used to configure multiple parallel threads and a summary thread, the parallel thread is used to parallelly process other operations other than the deduplication operation in the database operation statement, and the summary thread is used to summarize the execution results of the parallel threads and then perform the deduplication operation; then determines whether the parallel cost required to execute the parallel path is less than the serial cost required to execute the serial path; if so, selects the parallel path as the execution path of the database operation statement. Through this method, the deduplication keywords are eliminated first, and if the optimization cannot be eliminated, a serial path and a parallel path are created and the one with a lower cost is selected for execution, wherein since the deduplication keywords are difficult to be deduplicated in parallel, when the deduplication operation cannot be processed in parallel, the operations that can be processed in parallel are processed in parallel, and finally the deduplication is unified, thereby selecting the optimal execution path according to the actual situation of the database operation statement, and improving the deduplication efficiency of the database operation statement.

[0019] Furthermore, the method for processing database operation statements of the present invention eliminates duplicate keywords and executes the optimized database operation statements when the database operation statements meet the preset optimization conditions. Through this method, logical optimization operations such as eliminating duplicate keywords can be performed first, and then when the elimination is impossible, the execution path with the lowest cost is selected to execute the database operation statement, thereby maximizing the processing efficiency of the database operation statement.

[0020] Based on the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings, those skilled in the art will become more aware of the above and other objects, advantages and features of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Hereinafter, some specific embodiments of the present invention will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings: Figure 1 is a flowchart of a method for processing database operation statements according to an embodiment of the present invention; Figure 2 is a flowchart of a method for processing database operation statements according to another embodiment of the present invention; Figure 3 is a schematic diagram of a computer program product according to an embodiment of the present invention; Figure 4 is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention; and Figure 5 is a schematic diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] It should be understood by those skilled in the art that the embodiments described below are only some embodiments of the present invention, rather than all embodiments of the present invention, and these embodiments are intended to explain the technical principles of the present invention, rather than to limit the protection scope of the present invention. Based on the embodiments provided by the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should still fall within the protection scope of the present invention.

[0023] It should be noted that the logic and / or steps represented in the flowchart or described in other ways herein can be considered as a sequenced list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or equipment (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or equipment and execute instructions), or used in combination with these instruction execution systems, devices or equipment.

[0024] The present invention provides a method for processing database operation statements. Figure 1 FIG. 1 is a flow chart of a method for processing a database operation statement according to an embodiment of the present invention. Figure 1 As shown, the method for processing the database operation statement at least includes the following steps S101 to S106.

[0025] Step S101, obtaining database operation statements. An optional example is SQL (Structured Query Language). SQL statements have multiple functions such as data manipulation and data definition. This language has interactive characteristics and can provide great convenience for users. In this step, the way to receive database operation statements can generally be active input by the user, or automatically generated by the database program during operation due to business needs. Those skilled in the art can decide the way to receive database operation statements according to actual conditions.

[0026] Step S102, determine whether the database operation statement contains a deduplication keyword. In some optional embodiments, the deduplication keyword generally refers to distinct, which is a keyword in SQL used to remove duplicate rows from query results. When executing a query with distinct, the database system will compare each row of data in the result set. It will combine the values ​​of all specified columns of each row into a key value, and then check whether this key value has appeared in previous results. If it has not appeared, the row of data is included in the final result; if it has appeared, the row of data is ignored and the next row is processed.

[0027] In addition, before step S102, it generally includes: determining whether the database operation statement is a query statement; if so, executing the step of determining whether the database operation statement contains a deduplication keyword. The query statement is generally a select statement, combined with the deduplication keyword distinct, this type of query statement generally has a select distinct operation. For this type of database operation statement, in order to be able to use parallelism to improve the efficiency of deduplication, a logical conversion method is usually used to convert the distinct operation into a group by for parallel operation. However, in some special cases, the distinct operation cannot be converted into a group by, so the method of the present invention is to solve the problem that the conversion of distinct to group by faces some constraints and cannot be converted, resulting in the inability to achieve parallel deduplication.

[0028] Step S103, if the judgment in step S102 is yes, it is determined whether the database operation statement meets the preset optimization condition. The preset optimization condition is the condition required to eliminate duplicate keywords.

[0029] In the process of processing database operation statements, since distinct deduplication is relatively late and belongs to the physical optimization stage, this embodiment chooses to first perform logical optimization operations such as distinct elimination. If it is confirmed that the deduplication keyword distinct cannot be eliminated, then execute the steps of creating serial paths and parallel paths for database operation statements respectively. The improved optimization method can ensure the execution of optimization methods such as normal elimination or conversion of deduplication keywords. After confirming that it cannot be eliminated or converted, a path is created for distinct deduplication, and then the use of serial or parallel to process distinct deduplication is selected according to the result of cost estimation. This can greatly improve the processing efficiency of database operation statements.

[0030] Optionally, the step of judging whether the database operation statement meets the preset optimization conditions can generally include: determining the data column of the deduplication keyword operation in the database operation statement and the data table corresponding to the data column; judging whether the data column meets the non-emptiness condition and the uniqueness condition in the corresponding data table column by column; and judging whether the database operation statement meets the preset optimization conditions when all data columns meet the non-emptiness condition and the uniqueness condition in the corresponding data table. Among them, the non-emptiness condition and the uniqueness condition refer to the data column of the deduplication keyword operation due to the design of the data table itself or the limitation of the database operation statement. When the data column meets the non-emptiness condition and the uniqueness condition, the column data will inevitably not have duplicate data, so there is no need to perform deduplication operation. Therefore, when the database operation statement meets the preset optimization conditions, the deduplication keyword is eliminated, and the optimized database operation statement is executed. Thereby reducing the operations required for the database to process the database operation statement and improving the processing efficiency.

[0031] Step S104, if the judgment in step S103 is no, create a serial path and a parallel path for the database operation statement respectively. The serial path means that the tasks are executed in sequence, and the next operation is performed after one operation is completed. In the distinct deduplication operation of the database, the serial path means that the data is processed row by row or part by part to complete deduplication. For example, for a query result set containing 1,000 rows of data, when deduplication is performed serially, it will start from the first row and check row by row whether it is repeated with the previously processed rows until all rows are processed.

[0032] The parallel path executes multiple tasks at the same time. In the database environment, the system's multi-core processor, multi-threading and other technologies are used to divide the data into multiple parts and process these parts at the same time. Due to the complex semantics of distinct, it is difficult to parallelize it, so it is generally chosen to convert distinct into group by, which can simply process the data in each group and then merge the results. However, in some special cases, distinct cannot be converted into group by, such as the existence of aggregation functions, window functions, etc. In order to enable the above-mentioned database operation statements to be processed in parallel in the face of special cases such as aggregation functions and window functions, in some optional embodiments of the method of the present invention, the step of creating a parallel path for the database operation statement can generally include: determining the parallelism corresponding to the parallel path; establishing the same number of parallel threads according to the parallelism, and the parallel threads are used to parallelly process other operations other than the deduplication operation in the database operation statement; establishing a summary thread, and the summary thread is used to perform deduplication operations after summarizing the execution results of the parallel threads. In this way, parallel threads can be used as much as possible to improve data processing efficiency, and the parts that cannot be processed in parallel (deduplication operations) are then uniformly summarized and processed.

[0033] Step S105 , determining whether the parallel cost required to execute the parallel path is less than the serial cost required to execute the serial path.

[0034] In some optional embodiments, the step of determining whether the parallel cost required to execute the parallel path is less than the serial cost required to execute the serial path may generally include: estimating the serial cost required to execute the serial path and the parallel cost required to execute the parallel path.

[0035] The steps of estimating the serial cost required to execute the serial path can generally include: determining the type of operation to be performed, the data column to be operated, and the data table corresponding to the data column in the deduplication process according to the database operation statement; determining the amount of data to be executed and the complexity of the operation through the type of operation to be performed, the data column to be operated, and the data table corresponding to the data column; and calculating the serial cost based on the amount of data and the complexity of the operation.

[0036] The steps for estimating the parallel cost required to execute the parallel path can generally include: determining the type of operation to be performed, the data column to be operated, and the data table corresponding to the data column in the deduplication process according to the database operation statement; determining the amount of data to be executed and the complexity of the operation through the type of operation to be performed, the data column to be operated, and the data table corresponding to the data column; calculating the parallel thread cost corresponding to the execution of the parallel thread and the summary thread cost corresponding to the execution of the summary thread according to the amount of data and the complexity of the operation combined with the degree of parallelism; and combining the parallel thread cost and the summary thread cost as the parallel cost.

[0037] The time cost of the serial path mainly depends on the amount of data and the complexity of the operation. Suppose a distinct operation is performed on a table containing n rows of data. If it is a simple row-by-row comparison to remove duplicates (such as comparing each row to the previously processed row to see if it is the same), the time complexity is approximately For example, to remove duplicates in an unsorted list, each new element needs to be compared with all previous elements. But if the data is sorted, the time complexity can be optimized to , because the duplication can be quickly determined by comparing adjacent elements. Impact on operation complexity: If other operations are involved in the deduplication process, such as data type conversion and function calls (such as SUM, AVG and other aggregate functions), the time cost will increase accordingly. For example, before each comparison of elements, a complex function needs to be called to convert the data format, which will make each comparison operation take longer.

[0038] The resource cost of the serial path generally includes memory usage and CPU usage. For memory usage: In the serial deduplication process, the main memory consumption is used to store processed data (for comparison of duplication). If it is a simple storage of the data elements themselves, the memory consumption is roughly proportional to the amount of data. However, if additional data structures need to be stored (such as indexes and hash tables established to speed up comparisons), memory consumption will increase. For example, a hash table is used to store processed data for quick search of duplications, and the size of the hash table will increase as the amount of data increases. For CPU usage: The CPU is mainly used for data comparison and other operations (such as function calls). Its usage also depends on the amount of data and the complexity of the operation. For simple comparison operations, the CPU usage is relatively stable, but if the operation is complex (such as performing complex mathematical calculations on each row of data before comparison), the CPU usage will be higher.

[0039] The time cost of parallel paths generally includes data partitioning overhead, parallel task execution time, and result merging time. The time cost of data partitioning must be considered first in parallel paths. The time complexity of each part (assuming the parallelism is m) may be , depending on the partitioning algorithm. For example, simply partitioning the data by a fixed number of rows may be faster, but if the data is partitioned based on some complex attribute (such as partitioning by the hash value range of the data), the time cost will be higher.

[0040] Parallel task execution time: The time it takes for each parallel task to perform deduplication is similar to that of the serial path, but due to the increase in data size (assuming the data is evenly partitioned), so the time complexity of each task may be (unsorted data) or (Sorted data). However, this is only an ideal situation. In reality, the synchronization waiting time between parallel tasks also needs to be considered.

[0041] Result merging time: After completing the parallel tasks, the results of each part need to be merged and deduplicated in the summary thread. The merging time depends on the size of the result set and the complexity of the merging operation.

[0042] The resource cost of parallel paths generally includes: memory usage and CPU usage. In addition to the memory required by each parallel task to store data similar to the serial path, the parallel path may also require additional memory to store intermediate results and data partitioning information. The total memory consumption may be m times that of the serial path (with a parallelism of m) plus the memory used to store intermediate results and partitioning information.

[0043] CPU usage: As multiple tasks are executed in parallel, the total CPU usage will increase in a short period of time. However, it should be noted that parallelism does not always fully utilize CPU resources because there are still overheads such as task switching and synchronization.

[0044] In addition, there may be other factors that affect the cost estimation, such as the current load of the system, etc. Technical personnel in this field can decide how to calculate the serial cost and the parallel cost based on the actual situation of the database.

[0045] Step S106: If the answer in step S105 is yes, select the parallel path as the execution path of the database operation statement.

[0046] Among them, when the amount of data is small, the system resources are limited (such as a single-core processor or a device with small memory), or the task itself is relatively simple, the serial path may have better performance. Because it does not have the resource allocation and coordination overhead of parallel processing. When the amount of data is large and the system has sufficient parallel processing capabilities (such as multi-core processors, sufficient memory), the parallel path may be more efficient than the serial path. It can reduce the total processing time by processing multiple parts of data at the same time. However, the performance improvement of the parallel path is not linear and will be affected by factors such as the rationality of data division and system resource competition. Therefore, this method chooses to estimate the cost required by both parties and chooses the lowest cost method to process the database operation statement.

[0047] Through this method, deduplication keywords are eliminated first. If the elimination optimization cannot be performed, a serial path and a parallel path are created and the one with lower cost is selected for execution. Since deduplication keywords are difficult to deduplicate in parallel, when deduplication operations cannot be processed in parallel, operations that can be processed in parallel are processed in parallel, and finally deduplication is unified, so as to select the optimal execution path according to the actual situation of the database operation statement. When it is known that the parallel cost is lower than the serial cost (such as deduplication work with high data duplication), the parallel execution method can be used to the greatest extent to improve the deduplication efficiency of database operation statements.

[0048] Figure 2 FIG. 1 is a flow chart of a method for processing a database operation statement according to another embodiment of the present invention. Figure 2 As shown, the method for processing the database operation statement at least includes the following steps S201 to S211.

[0049] Step S201, obtaining a database operation statement.

[0050] Step S202, determine whether the database operation statement is a query statement. If the determination result is no, then execute step S211 to continue other operations of the optimizer.

[0051] Step S203, if the judgment result in step S202 is yes, it is judged whether the database operation statement contains the deduplication keyword distinct. If the judgment result is no, step S211 is executed.

[0052] Step S204, if the judgment in step S203 is yes, it is determined whether the database operation statement meets the preset optimization conditions. The preset optimization conditions are the conditions required to eliminate the deduplication keywords, and the specific judgment process can generally include: determining the data columns of the deduplication keyword operation in the database operation statement and the data table corresponding to the data columns; judging column by column whether the data columns meet the non-emptiness condition and the uniqueness condition in the corresponding data table; if all the data columns meet the non-emptiness condition and the uniqueness condition in the corresponding data table, it is determined that the database operation statement meets the preset optimization conditions.

[0053] Step S205, if the judgment in step S204 is yes, remove the deduplication keyword and execute the optimized database operation statement. If step S205 is executed, execute step S211. If the data column meets the non-empty condition and the uniqueness condition, there will be no duplicate data in the column data, so there is no need to perform deduplication operation. Therefore, in this case, you can choose to remove the deduplication keyword, thereby reducing the operations required for the database to process the database operation statement and improving processing efficiency.

[0054] Step S206, when the judgment in step S204 is no, a serial path and a parallel path are created for the database operation statement respectively. Among them, the serial path means that the tasks are executed in sequence, and the next operation is performed after one operation is completed. The parallel path executes multiple tasks at the same time, but due to the complex semantics of distinct, it is difficult to parallelize it. Generally, the database cannot directly create a parallel path for it. Therefore, the scheme for creating a parallel path of the present invention generally includes: determining the parallelism corresponding to the parallel path; establishing the same number of parallel threads according to the parallelism, and the parallel threads are used to parallelly process other operations other than the deduplication operation in the database operation statement; then establishing a summary thread, and the summary thread is used to summarize the execution results of the parallel threads and then perform the deduplication operation. In this way, parallel threads can be used as much as possible to improve data processing efficiency, and the parts that cannot be processed in parallel (deduplication operations) can be uniformly summarized and processed.

[0055] Step S207, estimate the serial cost required to execute the serial path and the parallel cost required to execute the parallel path. The cost generally includes time cost and resource cost, etc. The factors affecting the size of the cost mainly depend on the amount of data and the complexity of the operation. Therefore, in some optional embodiments, the method chooses to determine the type of operation to be performed, the data column to be operated, and the data table corresponding to the data column in the deduplication process according to the database operation statement; determine the amount of data to be performed and the complexity of the operation by the type of operation to be performed, the data column to be operated, and the data table corresponding to the data column; calculate the serial cost according to the amount of data and the complexity of the operation; calculate the parallel cost according to the amount of data and the complexity of the operation combined with the degree of parallelism.

[0056] In addition, there may be other factors that affect the cost estimation, such as the current load of the system, etc. Technical personnel in this field can decide how to calculate the serial cost and the parallel cost based on the actual situation of the database.

[0057] Step S208, determining whether the serial cost is greater than the parallel cost.

[0058] Step S209: If the result of step S208 is negative, the serial path is selected to perform distinct deduplication. In the case of small data volume, limited system resources (such as a single-core processor or a device with small memory), or a simple task, the serial path may have better performance because it does not have the resource allocation and coordination overhead of parallel processing.

[0059] Step S210: If the result of step S208 is yes, select a parallel path to perform distinct deduplication. When the amount of data is large and the system has sufficient parallel processing capabilities (such as multi-core processors and sufficient memory), the parallel path may be more efficient than the serial path. It can reduce the total processing time by processing multiple parts of data at the same time.

[0060] Step S211, continue other operations of the optimizer.

[0061] Through this method, logical optimization operations such as eliminating duplicate keywords can be performed first, and then if elimination is not possible, the execution path with the lowest cost is selected to execute the database operation statement, thereby maximizing the processing efficiency of the database operation statement.

[0062] In order to clearly illustrate the effect of this method, a specific embodiment is proposed as follows: select distinct sum(a) from distinct_test Formula 1 After receiving the SQL statement shown in Formula 1, the database meets the judgment conditions of step S202 and step S203 because it contains the select keyword and the distinct keyword, and then executes step S204. Since the sum(a) function exists inside it, the distinct keyword cannot be eliminated and cannot be converted into group by, so step S206 is continued to create a serial path and a parallel path for the database operation statement to process distinct deduplication, and then the cost consumed by the two paths is estimated, and the path with the lowest cost is selected for distinct deduplication.

[0063] The serial path sums the data column a row by row, and then removes duplicates from all the summed results. The parallel path establishes the same number of parallel threads according to the predetermined parallelism, and each parallel thread is only responsible for summing its own data. The summary thread established later is responsible for summing and removing duplicates from the processing results of the parallel threads.

[0064] This solution not only ensures the distinct elimination function, but also improves the data processing speed through parallelism as much as possible when distinct cannot be converted to group by.

[0065] In actual use, after applying the method of the present invention, the database can improve query efficiency by using parallelism. When facing the database operation statement shown in formula (1), in the scenario where the degree of parallelism is 4, the performance is improved by more than 75% by creating parallel paths.

[0066] The flow chart provided by the present embodiment is not intended to indicate that the operation of the method will be performed in any particular order, or that all operations of the method are included in all every case. In addition, the method may include additional operations. Within the scope of the technical thinking provided by the present embodiment method, additional changes may be made to the above method.

[0067] It should be understood that in some embodiments, each part can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system.

[0068] This embodiment also provides a computer program product 10 , a computer readable storage medium 20 , and a computer device 30 . Figure 3 is a schematic diagram of a computer program product 10 according to an embodiment of the present invention, Figure 4 is a schematic diagram of a computer-readable storage medium 20 according to an embodiment of the present invention, Figure 5 is a schematic diagram of a computer device 30 according to an embodiment of the present invention. The computer program product 10 includes a computer program 11, which implements the steps of any of the above-mentioned methods for processing database operation statements when executed by a processor 32. The computer-readable storage medium 20 stores the above-mentioned computer program 11, which implements the steps of any of the above-mentioned methods for processing database operation statements when executed by a processor 32. The computer device 30 may include a memory 31, a processor 32, and the computer program 11 stored in the memory 31 and running on the processor 32.

[0069] The computer program 11 for performing the operation of the present invention may be an assembly instruction, an instruction set architecture (ISA) instruction, a machine instruction, a machine-related instruction, a microcode, a firmware instruction, a state setting data, a configuration data of an integrated circuit, or a source code or an object code written in any combination of one or more programming languages ​​and process programming languages. The computer program 11 may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider through the Internet). In some embodiments, in order to perform various aspects of the present invention, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA) may execute computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit.

[0070] For the purposes of the description of this embodiment, the computer program product 10 is a related product that includes a computer program 11. For the purposes of the description of this embodiment, the computer-readable storage medium 20 is a tangible device capable of retaining and storing the computer program 11, which may be any device that can contain, store, communicate, propagate or transmit the program 11 for use in or in conjunction with an instruction execution system, device or apparatus. More specific examples (a non-exhaustive list) of the computer-readable storage medium 20 include the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, and any suitable combination of the above.

[0071] The computer device 30 may be, for example, a server, a desktop computer, a notebook computer, a tablet computer, or a smart phone. In some examples, the computer device 30 may be a cloud computing node. The computer device 30 may be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, a program module may include routines, programs, object programs, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. The computer device 30 may be implemented in a distributed cloud computing environment where remote processing devices linked through a communication network perform tasks. In a distributed cloud computing environment, program modules may be located on a local or remote computing system storage medium including a storage device.

[0072] The computer device 30 may include a processor 32 adapted to execute stored instructions, and a memory 31 providing temporary storage space for the operation of the instructions during operation. The processor 32 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 31 may include a random access memory (RAM), a read-only memory, a flash memory, or any other suitable storage system.

[0073] The computer device 30 may also include a network adapter / interface and an input / output (I / O) interface. The I / O interface allows data to be input and output with external devices that may be connected to the computer device. The network adapter / interface may provide communication between the computer device and a network, which is typically shown as a communication network.

[0074] At this point, those skilled in the art should recognize that, although multiple exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications that conform to the principles of the present invention can still be directly determined or derived based on the content disclosed in the present invention without departing from the spirit and scope of the present invention. Therefore, the scope of the present invention should be understood and recognized as covering all these other variations or modifications.

Claims

1. A method for processing a database operation statement, comprising: Obtaining the database operation statement; Determining whether the database operation statement contains a deduplication keyword; If so, determining whether the database operation statement meets a preset optimization condition, wherein the preset optimization condition is a condition required to eliminate the deduplication keyword; When the database operation statement does not meet the preset optimization conditions, a serial path and a parallel path are respectively created for the database operation statement, wherein the parallel path is used to configure a plurality of parallel threads and a summary thread, the parallel threads are used to process other operations in the database operation statement in parallel except the deduplication operation, and the summary thread is used to perform the deduplication operation after summarizing the execution results of the parallel threads; Determining whether a parallel cost required to execute the parallel path is less than a serial cost required to execute the serial path; If so, select the parallel path as the execution path of the database operation statement.

2. The method for processing database operation statements according to claim 1, wherein: The step of creating a parallel path for the database operation statement comprises: Determine the degree of parallelism corresponding to the parallel path; The same number of the parallel threads and the summary thread are established according to the parallelism.

3. The method for processing database operation statements according to claim 2, wherein: The step of determining whether the parallel cost required to execute the parallel path is less than the serial cost required to execute the serial path includes: estimating the serial cost required to execute the serial path and the parallel cost required to execute the parallel path; The step of estimating the serial cost required to execute the serial path includes: determining the type of operation to be performed, the data column to be operated, and the data table corresponding to the data column in the deduplication process according to the database operation statement; determining the amount of data to be executed and the complexity of the operation through the type of operation to be performed, the data column to be operated, and the data table corresponding to the data column; and calculating the serial cost based on the data amount and the operation complexity.

4. The method for processing database operation statements according to claim 3, wherein: The step of estimating the parallel cost required to execute the parallel path comprises: Determine, according to the database operation statement, the type of operation to be performed in the deduplication process, the data column to be operated, and the data table corresponding to the data column; Determine the amount of data to be executed and the complexity of the operation according to the type of operation to be executed, the data column to be operated and the data table corresponding to the data column; Calculate the parallel thread cost corresponding to executing the parallel thread and the summary thread cost corresponding to executing the summary thread according to the data volume and the operation complexity combined with the parallelism; The parallel thread cost and the summary thread cost are combined as the parallel cost.

5. The method for processing database operation statements according to claim 1, wherein: The step of determining whether the database operation statement meets the preset optimization condition comprises: Determine the data column of the deduplication keyword operation in the database operation statement and the data table corresponding to the data column; Determine column by column whether the data column meets the non-empty condition and the uniqueness condition in the corresponding data table; When all the data columns in the corresponding data table meet the non-emptiness condition and the uniqueness condition, it is determined that the database operation statement meets the preset optimization condition.

6. The method for processing database operation statements according to claim 5, wherein: The step of determining whether the database operation statement meets the preset optimization condition includes: When the database operation statement meets the preset optimization condition, the deduplication keyword is eliminated, and the optimized database operation statement is executed.

7. The method for processing database operation statements according to claim 1, wherein: Before the step of determining whether the database operation statement contains duplicate removal keywords, the following step is further included: Determining whether the database operation statement is a query statement; If so, execute the step of determining whether the database operation statement contains duplicate removal keywords.

8. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method for processing a database operation statement according to any one of claims 1 to 7 are implemented.

9. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method for processing a database operation statement according to any one of claims 1 to 7.

10. A computer device comprising a memory, a processor, and a machine executable program stored in the memory and running on the processor, wherein the processor implements the steps of the method for processing database operation statements according to any one of claims 1 to 7 when executing the machine executable program.

Images