Database parallel query insertion method and device, equipment and medium

Abstract

The invention discloses a database parallel query insertion method and device, equipment and a medium. The method comprises the following steps: acquiring an original query insertion statement with a histogram parallel instruction, and acquiring an equal-height histogram of a query table in the original query insertion statement; wherein the contour histogram is generated by a histogram statistics module in advance; generating a sub-screening condition according to the upper and lower limits of each storage bucket in the contour histogram and the parallel number of the parallel instruction in the original query insertion statement; splicing each sub-screening condition into a filtering condition of the original query insertion statement to obtain a plurality of sub-query insertion statements; and concurrently controlling the corresponding data nodes to execute the sub-query insertion statements, and determining the execution result of the original query insertion statement according to the execution results of all the sub-query insertion statements. The embodiment of the invention can improve the parallel query insertion performance of the database.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a database parallel query insertion method, device, equipment and medium. Background Art

[0002] Currently, parallel insertion into a database needs to be implemented with the help of an application or middleware layer according to the business. The application or middleware first divides the data to be inserted into multiple parts according to a certain strategy (such as primary key range, hash value, etc.), and then uses multiple threads or processes to write these data parts into the database at the same time. The existing parallel query insertion operation (insert select statement) requires modifying the application according to the business, or requires executing a query operation on the database first, processing the query results, and then executing the insertion operation. The efficiency of the database insertion operation is low. Summary of the invention

[0003] The present invention provides a database parallel query insertion method, device, equipment and medium to improve the performance of database parallel query insertion.

[0004] According to one aspect of the present invention, a database parallel query insertion method is provided, comprising:

[0005] Obtaining an original query insert statement with a histogram parallel instruction, and obtaining an iso-height histogram of the query table in the original query insert statement; wherein the iso-height histogram is generated in advance by a histogram statistics module;

[0006] Generate a sub-filtering condition according to the upper and lower limits of each storage bucket in the equal-height histogram and the parallel number of parallel instructions in the original query insert statement;

[0007] Each sub-filtering condition is spliced ​​into the filter condition of the original query insert statement to obtain multiple sub-query insert statements;

[0008] The corresponding data nodes are controlled in parallel to execute the sub-query insert statements, and the execution result of the original query insert statement is determined according to the execution results of all the sub-query insert statements.

[0009] According to another aspect of the present invention, there is provided a database parallel query insertion device, comprising:

[0010] A histogram acquisition module is used to acquire an original query insert statement with a histogram parallel instruction, and acquire an iso-height histogram of the query table in the original query insert statement; wherein the iso-height histogram is generated in advance by a histogram statistics module;

[0011] A sub-screening condition module, used to generate a sub-screening condition according to the upper and lower limits of each storage bucket in the equal-height histogram and the parallel number of parallel instructions in the original query insert statement;

[0012] A sub-insert statement module, used for splicing each sub-screening condition into the filter condition of the original query insert statement, to obtain multiple sub-query insert statements;

[0013] The statement execution module is used to control the corresponding data nodes to execute the sub-query insert statements in parallel, and determine the execution result of the original query insert statement according to the execution results of all the sub-query insert statements.

[0014] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the database parallel query insertion method according to any embodiment of the present invention.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the database parallel query insertion method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the database parallel query insertion method described in any embodiment of the present invention when executed.

[0020] The embodiment of the present invention uses the new histogram feature introduced into the database to generate sub-filtering conditions with the help of the histogram of the query table, and splices each sub-filtering condition into the filter condition of the original query insert statement to obtain multiple sub-query insert statements; the data nodes of the database are controlled in parallel to execute each sub-query insert statement, and the execution result of the original query insert statement is determined according to the execution result of the sub-query insert statement. Parallel query insertion is achieved with the help of the histogram function of the database, avoiding modification of the application. Parallel query insertion is achieved at the database level, which greatly improves the query insertion performance of the database and reduces the time and difficulty of SQL statement optimization. The flow of data between data nodes and computing nodes is avoided, and the communication pressure between nodes is reduced.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a first flow chart of a database parallel query insertion method provided by an embodiment of the present invention;

[0024] Figure 2 is a second flow chart of the database parallel query insertion method provided by an embodiment of the present invention;

[0025] Figure 3 is a third flow chart of the database parallel query insertion method provided by an embodiment of the present invention;

[0026] Figure 4 It is a structural diagram of a database parallel query insertion device provided by an embodiment of the present invention;

[0027] Figure 5 It is a schematic diagram of the structure of an electronic device implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Figure 1This is the first flow chart of the database parallel query insertion method provided by the embodiment of the present invention. This embodiment is applicable to the case where a database query insertion statement is executed in parallel through a histogram. The method can be executed by a database parallel query insertion device, which can be implemented in the form of hardware and / or software. The device can be configured in an electronic device with corresponding data processing capabilities. Figure 1 As shown, the method includes:

[0031] S110, obtaining an original query insert statement with a histogram parallel instruction, and obtaining an equal-height histogram of the query table in the original query insert statement; wherein the equal-height histogram is generated in advance by a histogram statistics module.

[0032] The query insert statement directly inserts the result of a query (select statement) into another table. The basic syntax of the query insert statement is:

[0033] INSERT INTO target_table(column1,column2,column3,...)

[0034] SELECT column1,column2,column3,...

[0035] FROM source_table

[0036] WHERE condition;

[0037] Among them, target_table is the table where data needs to be inserted; (column1, column2, column3, ...) are the names of the columns in target_table where data needs to be inserted; source_table is the table from which data is selected, that is, the query table; WHERE condition is used to filter the rows in source_table to determine which data is inserted into target_table. The query insertion statement is set according to the actual business needs, and this application does not make too many restrictions on this.

[0038] For databases that have introduced the histogram feature, such as MySQL 8.0, commands for directly generating histograms are supported. The histogram can be used to split data into multiple buckets of equal height. Each bucket has its upper and lower limits to mark the data range of the current grouped data, as well as the number of records contained in the current bucket.

[0039] The computing node of the distributed database obtains and parses the original query insert statement with the histogram parallel instruction, obtains the query table in the original query insert statement, and obtains the equal-height histogram of the query table. The equal-height histogram is used to generate the sub-filtering conditions of the query table. The original query insert statement is split into multiple sub-query insert statements according to the sub-filtering conditions, and distributed to the database data nodes for execution, so as to realize the efficient execution of the original query insert statement.

[0040] S120 , generating a sub-screening condition according to the upper and lower limits of each storage bucket in the equal-height histogram and the parallel number of parallel instructions in the original query insert statement.

[0041] The original query insert statement contains a histogram parallel instruction, which contains the maximum number of parallel executions of the original query insert statement. Determine the number of parallel executions specified in the parallel instruction of the original query insert statement, and obtain the upper and lower limits of each bucket in the equal-height histogram. Limit the data range of the query based on the upper and lower limits of the bucket and generate sub-filtering conditions. Using the histogram function to implement parallel query insertion can improve the query insertion performance of the database and reduce the time and difficulty of SQL statement optimization.

[0042] In an optional implementation, the sub-screening conditions do not overlap with each other, and all sub-screening conditions cover all values ​​in the specified column. By ensuring that the sub-screening conditions do not overlap with each other, and all sub-screening conditions cover all values ​​in the specified column, it helps to avoid duplicate processing or omission of data, while optimizing query efficiency.

[0043] S130: splice each sub-screening condition into the filter condition of the original query insert statement to obtain a plurality of sub-query insert statements.

[0044] For each sub-filtering condition, use it as the filter condition of the original query insert statement, and splice it into the filter condition of the original query insert statement to obtain the sub-query insert statement.

[0045] S140 , controlling the corresponding data nodes in parallel to execute the sub-query insert statements, and determining the execution result of the original query insert statement according to the execution results of all the sub-query insert statements.

[0046] Distribute each subquery insert statement to the data node of the database, and control the corresponding data node in parallel to execute each subquery insert statement. After receiving the execution results of all subquery insert statements, determine that the original query insert statement has been executed. Distribute each subquery insert condition to each data node for execution, avoiding the flow of data between data nodes and computing nodes, and reducing the communication pressure between nodes.

[0047] The embodiment of the present invention uses the new histogram feature introduced into the database to generate sub-filtering conditions with the help of the histogram of the query table, and splices each sub-filtering condition into the filter condition of the original query insert statement to obtain multiple sub-query insert statements; the data nodes of the database are controlled in parallel to execute each sub-query insert statement, and the execution result of the original query insert statement is determined according to the execution result of the sub-query insert statement. Parallel query insertion is achieved with the help of the histogram function of the database, avoiding modification of the application. Parallel query insertion is achieved at the database level, which greatly improves the query insertion performance of the database and reduces the time and difficulty of SQL statement optimization. The flow of data between data nodes and computing nodes is avoided, and the communication pressure between nodes is reduced.

[0048] In an optional implementation, the method further includes: generating a global transaction for the original query insert statement through a transaction manager; S140 includes: controlling the corresponding data nodes in parallel to execute each of the sub-query insert statements, and monitoring the execution status of each of the sub-query insert statements through the transaction manager; if each of the sub-query insert statements is completed, determining the execution result of the original query insert statement according to the execution result of each of the sub-query insert statements, and submitting the global transaction through the transaction manager to release resources associated with the global transaction.

[0049] Specifically, before the corresponding data nodes are controlled in parallel to execute each of the sub-query insert statements, a global transaction is generated for the original query insert statement through the transaction manager. The execution status of each sub-query insert statement is monitored through the transaction manager; if all sub-query insert statements are executed successfully, the global transaction is committed through the transaction manager, and the resources associated with the global transaction are released. If any sub-query insert statement fails to execute, the global transaction is rolled back through the transaction manager to undo all changes made in the transaction to ensure the consistency and integrity of the data. Through global transactions, not only the consistency and integrity of the data can be guaranteed, but also the reliability and stability of the system can be improved, the management of complex operations can be simplified, and the trust and satisfaction of users can be improved. The query conditions are spliced ​​in the computing nodes, and the sub-query insert conditions are distributed to each data node for execution, which avoids the flow of data between data nodes and computing nodes and reduces the communication pressure between nodes.

[0050] Figure 2 1 is a second flow chart of the database parallel query insertion method provided by an embodiment of the present invention. This embodiment is optimized and improved on the basis of the above embodiment. Figure 2 As shown, the method includes:

[0051] S210: Obtain the original query insert statement with the histogram parallel instruction.

[0052] S220, determining whether there is a histogram of equal heights of the query table in the original query insert statement in the database, if not, continuing to execute S230; if yes, reading the corresponding histogram of equal heights from the database.

[0053] S230 . Generate a corresponding equal-height histogram for a specified column in the query table according to the specified number of buckets.

[0054] The specified column can be any column in the query table. Optionally, if the original query insert statement contains a conditional judgment of the query table, the column in the query table conditional judgment can be selected as the specified column. You can also use a data column with dispersed data values ​​in the query table as the specified column, that is, the data values ​​of the column are widely distributed in the data set, or there are many different unique values, which is convenient for dividing the data in the query table into buckets with roughly the same number of data points, so as to achieve the balance of subsequent execution tasks of each database data node, which can not only improve the overall execution performance of the original query insert statement, but also reduce the risk of performance bottlenecks caused by data skew. The number of specified buckets can refer to the parallel number setting. For example, when the specified column contains null values, the number of specified buckets can be set to the parallel number minus one. When the specified column does not contain null values, the number of specified buckets can be set to the parallel number, so that the sub-filtering conditions can be generated according to the parallel number and the upper and lower limits of the buckets.

[0055] In an optional embodiment, the method of generating a corresponding equal-height histogram for a specified column in the query table according to a specified number of buckets includes: continuously dividing the specified column in the query table into equal-height partitions according to a specified number of buckets to obtain multiple groups of data, so that the amount of data in each group of data is relatively uniform; allocating the boundary values ​​of each group of data to the buckets as the upper and lower limits of the buckets to generate a equal-height histogram for the query table.

[0056] Optionally, for databases that support built-in functions or commands for directly generating histograms, a histogram is generated by directly specifying the columns and the number of buckets. For databases that do not directly provide built-in functions or commands for generating histograms, a histogram can be generated through statistical functions in the database. Through the new histogram feature introduced in the database, sub-filtering conditions are generated with the help of the histogram of the query table, and each sub-filtering condition is spliced ​​into the filter conditions of the original query insert statement to obtain multiple sub-query insert statements. Parallel query insertion is achieved with the help of the histogram function of the database, avoiding modifications to the application. Parallel query insertion is implemented at the database level, which greatly improves the query insertion performance of the database and reduces the time and difficulty of SQL statement optimization.

[0057] S240 , generating a sub-screening condition according to the upper and lower limits of each storage bucket in the equal-height histogram and the parallel number of parallel instructions in the original query insert statement.

[0058] S250: splice each sub-screening condition into the filter condition of the original query insert statement to obtain a plurality of sub-query insert statements.

[0059] S260 , controlling the corresponding data nodes in parallel to execute the sub-query insert statements, and determining the execution result of the original query insert statement according to the execution results of all the sub-query insert statements.

[0060] The embodiment of the present invention uses the new histogram feature introduced into the database to generate sub-filtering conditions with the help of the histogram of the query table, and splices each sub-filtering condition into the filter condition of the original query insert statement to obtain multiple sub-query insert statements; the data nodes of the database are controlled in parallel to execute each sub-query insert statement, and the execution result of the original query insert statement is determined according to the execution result of the sub-query insert statement. Parallel query insertion is achieved with the help of the histogram function of the database, avoiding modification of the application. Parallel query insertion is achieved at the database level, which greatly improves the query insertion performance of the database and reduces the time and difficulty of SQL statement optimization. The flow of data between data nodes and computing nodes is avoided, and the communication pressure between nodes is reduced.

[0061] Figure 3 FIG. 3 is a third flow chart of a database parallel query insertion method provided by an embodiment of the present invention. This embodiment is optimized and improved on the basis of the above embodiment. Figure 3 As shown, the method includes:

[0062] S310, obtaining an original query insert statement with a histogram parallel instruction, and obtaining an equal-height histogram of the query table in the original query insert statement; wherein the equal-height histogram is generated in advance by a histogram statistics module.

[0063] S320 , compare the number of buckets in the histogram with the parallel number. If the number of buckets is less than or equal to the parallel number, continue to execute S330 ; otherwise, jump to execute S340 .

[0064] Specifically, the number of sub-filtering conditions is determined based on the number of parallelism and the number of buckets in the equal-height histogram.

[0065] S330, taking the upper and lower limits of each storage bucket as a set of conditional boundary values ​​to generate a sub-screening condition.

[0066] Each bucket has its upper and lower limits to mark the data range of the current grouped data. Based on the upper and lower limits of the bucket, a sub-filtering condition can be generated to filter out records that fall within a specific bucket range from the data set. For example, the sub-filtering condition is where columnN between 12and320; columnN is the specified column set when the query table generates a histogram, 12 is the lower limit of the current bucket, and 320 is the upper limit of the current bucket.

[0067] S340: Merge two adjacent storage buckets to generate new upper and lower limits as a set of conditional boundary values ​​to generate a sub-screening condition.

[0068] When the number of buckets is greater than the number of parallel operations, two adjacent buckets can be merged to generate new upper and lower limits as a set of conditional boundary values. For example, when the number of parallel operations is 100 and the number of buckets is 200, the upper and lower limits of the first and second buckets can be merged to obtain new upper and lower limits as a set of conditional boundary values; the upper and lower limits of the third and fourth buckets can be merged to obtain new upper and lower limits as a set of conditional boundary values; and so on, until the conditional boundary values ​​of all generated groups cover the upper and lower limits of all buckets, and the number of conditional boundary value groups is the same as the number of parallel operations, and a sub-filter condition is generated according to each group of conditional boundary values.

[0069] Optionally, multiple adjacent buckets can be split again to obtain new upper and lower limits. For example, 5 adjacent buckets can be split again into 2 buckets to obtain 2 sets of conditional boundary values ​​and generate 2 sub-filtering conditions. When the number of buckets is greater than the number of parallel operations, the buckets can be split again according to the distribution of the data to ensure that the number of generated sub-filtering conditions is not greater than the number of parallel operations. This application does not make any detailed restrictions on this.

[0070] Based on the number of parallelism and the number of buckets in the equal-height histogram, the number of sub-filtering conditions is determined to ensure that the number of generated sub-query insert statements does not exceed the number of parallelism of the parallel instructions in the original query insert statement, and that each sub-query insert statement is executed with a reasonable degree of parallelism to ensure balanced resource allocation, avoid resource overload, reduce concurrency conflicts, simplify transaction management, and make performance predictable.

[0071] In an optional implementation, the method further includes: if a designated column for generating a histogram contains a null value, taking the value of the designated column as a null value as a sub-filtering condition, and reducing the parallel number by one.

[0072] Specifically, when generating a histogram of a query table, if the specified column contains null values, the statistical information of the null values ​​of the specified column is not included in the histogram. Therefore, the null value of the specified column is used as a sub-filtering condition, and the corresponding parallel number is reduced by 1. By using the null value of the specified column as a sub-filtering condition, it is ensured that records with no value in the column (i.e., the value is null) will not be missed when filtering data, thereby ensuring the correctness of the execution result of the original query insert statement.

[0073] S350: splice each sub-screening condition into the filter condition of the original query insert statement to obtain a plurality of sub-query insert statements.

[0074] S360, controlling the corresponding data nodes in parallel to execute the sub-query insert statements, and determining the execution result of the original query insert statement according to the execution results of all the sub-query insert statements.

[0075] The embodiment of the present invention determines the condition boundary value according to the upper and lower limits of the buckets by comparing the quantitative relationship between the number of buckets and the number of parallelism in the histogram, and then generates sub-filtering conditions, and splices each sub-filtering condition into the filter condition of the original query insert statement, thereby obtaining multiple sub-query insert statements. It ensures that the number of sub-query insert statements does not exceed the parallel number of parallel instructions in the original query insert statement, ensures that each sub-query insert statement is executed with a reasonable degree of parallelism, ensures balanced allocation of resources, avoids resource overload, reduces concurrency conflicts, simplifies transaction management, and makes performance predictable. Parallel query insertion is achieved with the help of the histogram function of the database, avoiding modifications to the application. Parallel query insertion is achieved at the database level, which greatly improves the query insertion performance of the database and reduces the time and difficulty of SQL statement optimization. It avoids the flow of data between data nodes and computing nodes, and reduces the communication pressure between nodes.

[0076] Figure 4 is a schematic diagram of the structure of a database parallel query insertion device provided by an embodiment of the present invention, such as Figure 4 As shown, the device comprises:

[0077] The histogram acquisition module 410 is used to acquire the original query insert statement with the histogram parallel instruction, and acquire the equal-height histogram of the query table in the original query insert statement; wherein the equal-height histogram is generated in advance by the histogram statistics module;

[0078] The sub-filtering condition module 420 is used to generate a sub-filtering condition according to the upper and lower limits of each storage bucket in the equal-height histogram and the parallel number of the parallel instructions in the original query insert statement;

[0079] The sub-insert statement module 430 is used to splice each sub-filtering condition into the filter condition of the original query insert statement to obtain multiple sub-query insert statements;

[0080] The statement execution module 440 is used to control the corresponding data nodes to execute the sub-query insert statements in parallel, and determine the execution result of the original query insert statement according to the execution results of all the sub-query insert statements.

[0081] The database parallel query insertion device provided in the embodiment of the present invention can execute the database parallel query insertion method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0082] Optionally, the histogram acquisition module includes:

[0083] The histogram generating unit is used to generate a corresponding equal-height histogram for a specified column in the query table according to a specified number of buckets if the equal-height histogram of the query table in the original query insert statement does not exist in the database.

[0084] The histogram generating unit is specifically used for:

[0085] Dividing the specified column in the query table into consecutive equal-height partitions according to the specified number of storage buckets to obtain multiple groups of data, so that the amount of data in each group of data is relatively uniform;

[0086] The boundary values ​​of each group of data are allocated to the storage buckets as the upper and lower limits of the storage buckets, and an equal-height histogram of the query table is generated.

[0087] Optionally, the sub-filtering conditions do not overlap with each other, and all sub-filtering conditions cover all values ​​in the specified column.

[0088] Optionally, the sub-screening condition module is specifically used to:

[0089] If the number of buckets in the histogram is less than or equal to the number of parallel operations, the upper and lower limits of each bucket are used as a set of conditional boundary values ​​to generate a sub-filtering condition;

[0090] If the number of buckets in the histogram is greater than the parallel number, two adjacent buckets are merged to generate new upper and lower limits as a set of conditional boundary values ​​to generate a sub-filtering condition.

[0091] Optionally, the sub-screening condition module includes:

[0092] The sub-screening condition obtaining unit is used to, if the designated column for generating the histogram contains a null value, take the value of the designated column as a null value as a sub-screening condition, and reduce the parallel number by one.

[0093] Optionally, the device further comprises:

[0094] The transaction module is used to generate a global transaction for the original query insert statement through the transaction manager.

[0095] Accordingly, the statement execution module is specifically used for:

[0096] The corresponding data nodes are controlled in parallel to execute each of the sub-query insert statements, and the execution status of each of the sub-query insert statements is monitored through the transaction manager; if each of the sub-query insert statements is completed, the execution result of the original query insert statement is determined according to the execution result of each of the sub-query insert statements, and the global transaction is committed through the transaction manager to release resources associated with the global transaction.

[0097] The database parallel query insertion device further described can also execute the database parallel query insertion method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0098] Figure 5 A schematic diagram of an electronic device 50 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0099] like Figure 5 As shown, the electronic device 50 includes at least one processor 51, and a memory connected to the at least one processor 51 in communication, such as a read-only memory (ROM) 52, a random access memory (RAM) 53, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 51 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 52 or the computer program loaded from the storage unit 58 to the random access memory (RAM) 53. In the RAM 53, various programs and data required for the operation of the electronic device 50 can also be stored. The processor 51, the ROM 52, and the RAM 53 are connected to each other via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0100] A number of components in the electronic device 50 are connected to the I / O interface 55, including: an input unit 56, such as a keyboard, a mouse, etc.; an output unit 57, such as various types of displays, speakers, etc.; a storage unit 58, such as a disk, an optical disk, etc.; and a communication unit 59, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 59 allows the electronic device 50 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0101] The processor 51 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 51 executes the various methods and processes described above, such as a database parallel query insertion method.

[0102] In some embodiments, the database parallel query insertion method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 58. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 50 via the ROM 52 and / or the communication unit 59. When the computer program is loaded into the RAM 53 and executed by the processor 51, one or more steps of the database parallel query insertion method described above may be performed. Alternatively, in other embodiments, the processor 51 may be configured to execute the database parallel query insertion method in any other appropriate manner (e.g., by means of firmware).

[0103] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0105] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0106] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0107] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0108] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0109] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0110] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A database parallel query insertion method, characterized in that: The method comprises: Obtaining an original query insert statement with a histogram parallel instruction, and obtaining an iso-height histogram of the query table in the original query insert statement; wherein the iso-height histogram is generated in advance by a histogram statistics module; Generate a sub-filtering condition according to the upper and lower limits of each storage bucket in the equal-height histogram and the parallel number of parallel instructions in the original query insert statement; Each sub-filtering condition is spliced ​​into the filter condition of the original query insert statement to obtain multiple sub-query insert statements; The corresponding data nodes are controlled in parallel to execute the sub-query insert statements, and the execution result of the original query insert statement is determined according to the execution results of all the sub-query insert statements.

2. The method according to claim 1, characterized in that The obtaining of the equal height histogram of the query table in the original query insert statement includes: If the equal-height histogram of the query table in the original query insert statement does not exist in the database, a corresponding equal-height histogram is generated for the specified column in the query table according to the specified number of buckets.

3. The method according to claim 2, characterized in that Generating a corresponding equal-height histogram for a specified column in the query table according to the specified number of buckets includes: Dividing the specified column in the query table into consecutive equal-height partitions according to the specified number of storage buckets to obtain multiple groups of data, so that the amount of data in each group of data is relatively uniform; The boundary values ​​of each group of data are allocated to the storage buckets as the upper and lower limits of the storage buckets, and an equal-height histogram of the query table is generated.

4. The method according to claim 1, characterized in that: The sub-filtering conditions do not overlap with each other, and all sub-filtering conditions cover all values ​​in the specified column.

5. The method according to claim 1 or 4, characterized in that: The sub-screening condition is generated according to the upper and lower limits of each storage bucket in the equal height histogram and the parallel number of parallel instructions in the original query insert statement, including: If the number of buckets in the histogram is less than or equal to the number of parallel operations, the upper and lower limits of each bucket are used as a set of conditional boundary values ​​to generate a sub-filtering condition; If the number of buckets in the histogram is greater than the parallel number, two adjacent buckets are merged to generate new upper and lower limits as a set of conditional boundary values ​​to generate a sub-filtering condition.

6. The method according to claim 1 or 4, characterized in that: The method further comprises: If the designated column for generating the histogram contains a null value, the designated column is set to a null value as a sub-filtering condition, and the parallel number is reduced by one.

7. The method according to claim 1, characterized in that The method further comprises: generating a global transaction for the original query insert statement through a transaction manager; The parallel control corresponding data node executes each of the sub-query insert statements, and determines the execution result of the original query insert statement according to the execution results of all the sub-query insert statements, including: Controlling the corresponding data nodes in parallel to execute each of the sub-query insert statements, and monitoring the execution status of each of the sub-query insert statements through the transaction manager; If all the sub-query insert statements are completed, the execution result of the original query insert statement is determined according to the execution results of each sub-query insert statement, and the global transaction is committed through the transaction manager to release resources associated with the global transaction.

8. A database parallel query insertion device, characterized in that: The device comprises: A histogram acquisition module is used to acquire an original query insert statement with a histogram parallel instruction, and acquire an iso-height histogram of the query table in the original query insert statement; wherein the iso-height histogram is generated in advance by a histogram statistics module; A sub-screening condition module, used to generate a sub-screening condition according to the upper and lower limits of each storage bucket in the equal-height histogram and the parallel number of parallel instructions in the original query insert statement; A sub-insert statement module, used for splicing each sub-screening condition into the filter condition of the original query insert statement to obtain multiple sub-query insert statements; The statement execution module is used to control the corresponding data nodes to execute the sub-query insert statements in parallel, and determine the execution result of the original query insert statement according to the execution results of all the sub-query insert statements.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the database parallel query insertion method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the database parallel query insertion method according to any one of claims 1 to 7 when executed.

Images