Data processing method and device, equipment and storage medium

Abstract

This application discloses a data processing method, apparatus, device, and storage medium, relating to the field of database technology. It is used to automatically determine whether to perform dynamic filtering on SQL query statements, helping to reduce the manual cost of performing dynamic filtering and avoiding "negative speedup." The method includes: receiving a Structured Query Language (SQL) query statement and determining the execution information of the SQL query statement; the execution information includes at least one of the following: multi-table join information of at least two tables, query condition information of the SQL query statement, and query field information of the SQL query statement; predicting a first duration and a second duration based at least on the execution information; wherein the first duration is the execution time of the SQL query statement when dynamic filtering is performed, and the second duration is the execution time of the SQL query statement when dynamic filtering is not performed; and performing dynamic filtering on the SQL query statement if the first duration is less than or equal to the second duration.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a data processing method, apparatus, device and storage medium. Background Technology

[0002] When a Structured Query Language (SQL) query requires querying at least two tables, and two of these tables are related, a multi-table join operation is typically performed. Furthermore, dynamic filtering can be enabled during multi-table joins to pre-filter the larger table in the join, reducing data loading and improving overall query performance. Specifically, this is usually done manually by operations personnel using configuration options, setting a preset duration. When the time taken to read the smaller table in the join is less than or equal to the preset duration, filter conditions are generated, and the larger table in the join is dynamically filtered based on these conditions.

[0003] In the above solution, the need for operations and maintenance personnel to manually enable the database's dynamic filtering function and set a preset duration results in high labor costs and places high demands on these personnel. Furthermore, manually setting the preset duration has low accuracy; if the time taken to read a smaller table exceeds the preset duration, filtering conditions cannot be generated, meaning dynamic filtering cannot be executed. In this case, because the large table is only read after the preset duration, it negatively impacts the execution time of SQL queries. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, and storage medium for automatically determining whether to perform dynamic filtering on SQL query statements, which helps reduce the manual cost of performing dynamic filtering and avoids causing "negative acceleration".

[0005] To achieve the above objectives, the embodiments of this application provide the following technical solutions:

[0006] In a first aspect, a data processing method is provided for a computer device, comprising: receiving a Structured Query Language (SQL) query statement and determining execution information of the SQL query statement; the SQL query statement is used to query at least two tables in a database, and the execution information includes at least one of the following: multi-table join information of the at least two tables, query condition information of the SQL query statement, and query field information of the SQL query statement; predicting a first duration and a second duration based at least on the execution information; wherein the first duration is the execution duration of the SQL query statement when dynamic filtering is performed on the SQL query statement, and the second duration is the execution duration of the SQL query statement when dynamic filtering is not performed on the SQL query statement; and performing dynamic filtering on the SQL query statement when the first duration is less than or equal to the second duration.

[0007] In this solution, for each received SQL query, a first execution time and a second execution time are determined. Dynamic filtering is performed on the SQL query if the first execution time is less than or equal to the second execution time. On one hand, this achieves automatic determination of whether to perform dynamic filtering on SQL queries, enabling adaptive dynamic filtering and reducing the manual cost of dynamic filtering, thus addressing the issue of high requirements for maintenance personnel in related technologies. On the other hand, since dynamic filtering is only performed on SQL queries when the first execution time is less than or equal to the second execution time, and not performed when the first execution time is greater than the second execution time, the solution with the shortest execution time is selected. Therefore, the optimal solution is chosen in the determination of whether to perform dynamic filtering, thus avoiding the problem of dynamic filtering causing query performance degradation when the time saved by performing dynamic filtering is insufficient to compensate for the time spent generating the filtering conditions. Furthermore, since no preset execution time needs to be set, it also solves the problem of "negative acceleration" that is common in related technologies.

[0008] In one possible implementation, the multi-table join information includes at least one of the following: the total number of multi-table joins in at least two tables, the number of inner joins, the number of left joins, the number of right joins, the number of full joins, the number of cross joins, the number of hash-type multi-table joins, the number of merge-type multi-table joins, and the number of nested multi-table joins; and / or the query condition information includes the multi-table join status of the query fields in the SQL query statement and / or the size relationship between the column data corresponding to the query fields and the filter conditions; the multi-table join status is used to characterize whether the tables corresponding to the query fields are performing multi-table joins; and / or the query field information includes at least one of the following: the number of rows, average value, number of distinct values, number of null values, maximum value, and minimum value of the column data corresponding to the query fields in the SQL query statement.

[0009] This possible implementation provides specific information including multi-table join information, query field information, and query condition information. If the first duration and the second duration are predicted based on this specific information, it will help improve the accuracy of the predicted first duration and the second duration.

[0010] In one possible implementation, predicting the first duration and the second duration based at least on execution information includes: predicting the first duration and the second duration based on execution information and configuration information of the computer device; the configuration information includes at least one of the following: hardware environment information of the computer device, software environment information of the computer device, and current operating environment information of the computer device.

[0011] This possible implementation method, based on execution information, comprehensively considers the impact of computer equipment on the execution time of SQL query statements, which helps to improve the accuracy of the predicted first and second execution times.

[0012] In one possible implementation, predicting a first duration and a second duration based on execution information includes: determining a target feature vector, which is used to represent the execution information; predicting a first duration based on the target feature vector and a first duration prediction model; the first duration prediction model is used to predict the execution duration of a SQL query statement represented by a feature vector that performs dynamic filtering; predicting a second duration based on the target feature vector and the second duration prediction model; and the second duration prediction model is used to predict the execution duration of a SQL query statement represented by a feature vector that does not perform dynamic filtering.

[0013] This possible implementation provides a specific way to predict the first and second durations, which is simple, convenient, efficient, and helps to improve the accuracy of the predicted first and second durations.

[0014] In one possible implementation, the method further includes: acquiring training samples, which include historical execution information of historical SQL query statements; the historical SQL query statements are used to query at least two sample tables in the database, and the historical execution information includes at least one of the following: historical multi-table join information of at least two sample tables, historical query condition information of historical SQL query statements, and historical query field information of historical SQL query statements; determining sample feature vectors, which are used to characterize the historical execution information; training a linear regression model based on the sample feature vectors and a first sample duration to obtain a first duration prediction model; the first sample duration is the execution duration of historical SQL query statements when dynamic filtering is performed on the historical SQL query statements; training a linear regression model based on the sample feature vectors and a second sample duration to obtain a second duration prediction model; the second sample duration is the execution duration of historical SQL query statements when dynamic filtering is not performed on the historical SQL query statements.

[0015] This possible implementation method, by training a linear regression model using historical execution information of historical SQL query statements, the duration of the first sample, and the duration of the second sample, helps to improve the accuracy of the first duration prediction model and the second duration prediction model in predicting the first duration and the second duration.

[0016] In a second aspect, a data processing apparatus is provided, comprising: functional units for executing any of the methods provided in the first aspect, wherein the actions performed by each functional unit are implemented by hardware or by hardware executing corresponding software. For example, a computer device may include: an acquisition unit, a prediction unit, and an execution unit; the acquisition unit is configured to receive a Structured Query Language (SQL) query statement and determine the execution information of the SQL query statement; the SQL query statement is used to query at least two tables in a database; the execution information includes at least one of the following: multi-table join information of at least two tables, query condition information of the SQL query statement, and query field information of the SQL query statement; the prediction unit is configured to predict a first duration and a second duration based at least on the execution information; wherein the first duration is the execution duration of the SQL query statement when dynamic filtering is performed on the SQL query statement, and the second duration is the execution duration of the SQL query statement when dynamic filtering is not performed on the SQL query statement; the execution unit is configured to perform dynamic filtering on the SQL query statement when the first duration is less than or equal to the second duration.

[0017] Thirdly, a computer device is provided, comprising: a processor and a memory. The processor is connected to the memory, the memory being used to store computer execution instructions, and the processor executing the computer execution instructions stored in the memory, thereby implementing any of the methods provided in the first aspect.

[0018] Fourthly, a chip is provided, comprising: a processor and an interface circuit; the interface circuit for receiving code instructions and transmitting them to the processor; and the processor for executing the code instructions to perform any of the methods provided in the first aspect.

[0019] Fifthly, a computer-readable storage medium is provided, storing computer-executable instructions that, when executed on a computer, cause the computer to perform any of the methods provided in the first aspect.

[0020] In a sixth aspect, a computer program product is provided, including computer execution instructions that, when executed on a computer, cause the computer to perform any of the methods provided in the first aspect.

[0021] The technical effects of any of the implementation methods in aspects two through six can be found in the technical effects of the corresponding implementation methods in aspect one, and will not be repeated here. Attached Figure Description

[0022] Figure 1 A schematic diagram of the structure of a computer device provided in an embodiment of this application;

[0023] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0024] Figure 3 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0025] Figure 4 A schematic diagram of a data processing method provided in an embodiment of this application;

[0026] Figure 5 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0027] Figure 6 A schematic diagram of another data processing method provided in an embodiment of this application;

[0028] Figure 7 A schematic diagram of another data processing method provided in an embodiment of this application;

[0029] Figure 8 A schematic diagram of another data processing method provided in an embodiment of this application;

[0030] Figure 9 This is a schematic diagram of the composition of a data processing device provided in an embodiment of this application. Detailed Implementation

[0031] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0032] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0033] First, to facilitate understanding of this application, the relevant elements involved in this application will now be described.

[0034] (1) Massively Parallel Processing (MPP): This involves distributing tasks in parallel across multiple nodes (usually servers). After computation is completed on each node, the results of each part are combined to obtain the final result.

[0035] (2) Dynamic Filtering: Also known as Runtime Filtering, it is an optimization technique widely used in MPP databases. Its basic principle is to filter out input data that will not be included in the join by filtering it out in advance on the probe side of the multi-table join (the probe side is usually the larger table in the multi-table join), so as to greatly reduce the data transfer and calculation in the join, thereby reducing the overall execution time.

[0036] Secondly, a brief introduction to the application scenarios involved in this application will be given.

[0037] In relational database management systems, such as MPP database management systems, the relationships between data in a table do not need to be defined when the table is created; often, all information about a single entity is stored in one table. When retrieving data, information about different entities stored in multiple tables can be retrieved using multi-table join operations. Multi-table join operations provide users with great flexibility, allowing them to add new data types at any time. This involves creating new tables for different entities and then querying them using multi-table join operations.

[0038] Furthermore, while multi-table join operations can increase the convenience of data querying, they are also time-consuming and resource-intensive operations, especially the shuffle operation involved, which consumes significant resources and time in massive data scenarios. For shuffle operations, the cloud-native big data computing service (MaxCompute, formerly known as ODPS) can optimize them using the equi-join property of JOIN. Specifically, based on the characteristics of JOIN equi-join, MaxCompute can generate a filter from the data in table A (the smaller table) to pre-filter the data in table B (the larger table) before shuffling or joining, i.e., dynamic filtering.

[0039] Next, a brief introduction to the implementation environment (implementation architecture) involved in this application will be given below.

[0040] This application provides a data processing method applicable to computer equipment. This application does not limit the specific form of the computer equipment. For example, the computer equipment can be a terminal device or a network device. The terminal device can be referred to as: terminal, user equipment (UE), terminal device, access terminal, user unit, user station, mobile station, remote station, remote terminal, mobile device, user terminal, wireless communication equipment, user agent, or user equipment, etc. Specifically, the terminal device can be a mobile phone, augmented reality (AR) device, virtual reality (VR) device, tablet computer, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc. The network device can be a server, etc. The server can be a single physical or logical server, or two or more physical or logical servers sharing different responsibilities and cooperating to achieve the various functions of the server.

[0041] In terms of hardware implementation, the aforementioned computer equipment can be implemented through, for example... Figure 1 The computer device shown is implemented as follows. Figure 1 The diagram shown is a hardware structure schematic of a computer device 100 provided in an embodiment of this application. The computer device 100 can be used to implement the functions of the aforementioned computer device.

[0042] Figure 1 The computer device 100 shown may include a processor 101, a memory 102, a communication interface 103, and a bus 104. The processor 101, the memory 102, and the communication interface 103 can be connected via the bus 104.

[0043] The processor 101 is the control center of the computer device 100. It can be a general-purpose central processing unit (CPU) or other general-purpose processors. The general-purpose processor can be a microprocessor or any conventional processor.

[0044] As an example, processor 101 may include one or more CPUs, for example Figure 1 CPU 0 and CPU 1 are shown in the diagram.

[0045] The memory 102 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0046] In one possible implementation, the memory 102 may exist independently of the processor 101. The memory 102 can be connected to the processor 101 via a bus 104 and is used to store data, instructions, or program code. When the processor 101 calls and executes the instructions or program code stored in the memory 102, it can implement the vehicle abnormal behavior detection method provided in this application embodiment.

[0047] In another possible implementation, the memory 102 can also be integrated with the processor 101.

[0048] The communication interface 103 is used for the computer device 100 to connect with other devices via a communication network, which may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The communication interface 103 may include a receiving unit for receiving data and a transmitting unit for transmitting data.

[0049] Bus 104 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 1 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0050] It should be pointed out that, Figure 1 The structure shown does not constitute a limitation on the computer device 100, except... Figure 1In addition to the components shown, the computer device 100 may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0051] like Figure 2 The diagram shown is a flowchart of a data processing method provided in this application, used in a computer device 100. The method includes:

[0052] S201: Receives a Structured Query Language (SQL) query statement and determines the execution information of the SQL query statement.

[0053] The SQL query statement is used to query at least two tables in the database. Execution information includes at least one of the following: multi-table join information for at least two tables, query conditions for the SQL query statement, and query field information for the SQL query statement.

[0054] Optionally, an SQL query statement includes a table name, query fields, and query conditions. The table name indicates the name of the table to be queried, the query fields indicate the column data in the table, and the query conditions are used to filter the list data.

[0055] In one implementation, the SQL query statement includes person (table name), age (query field), and less than 20 (query condition). For example, if the query field indicates a column with a minimum value of 15 and a maximum value of 30, the query condition will filter out values ​​greater than or equal to 20.

[0056] In one implementation, after receiving an SQL query statement, the SQL query statement is parsed to obtain the execution information of the SQL query statement.

[0057] Optionally, the multi-table join information includes at least one of the following: the total number of multi-table joins in at least two tables, the number of inner joins, the number of left joins, the number of right joins, the number of full joins, the number of cross joins, the number of hash joins, the number of merge joins, and the number of nested joins.

[0058] By defining the number of different types of multi-table joins and the total number of multi-table joins as belonging to multi-table join information, the correlation between multi-table join information and the time required to perform dynamic filtering is improved, which in turn helps to improve the accuracy of the first and second predicted durations.

[0059] In one implementation, for example, at least two tables include Table 1, Table 2, Table 3, Table 4, Table 5, Table 6, Table 7, Table 8, Table 9, and Table 10. Table 1 and Table 2 perform multi-table joins (specifically, inner joins, hash-type joins); Table 3 and Table 4 perform multi-table joins (specifically, left joins, merge-type multi-table joins); Table 5 and Table 6 perform multi-table joins (specifically, right joins, nested-type multi-table joins); Table 7 and Table 8 perform multi-table joins (specifically, full joins, merge-type multi-table joins); and Table 9 and Table 10 perform multi-table joins (specifically, cross joins, merge-type multi-table joins). Therefore, in at least two tables, the total number of multi-table joins is 5, the number of inner joins is 1, the number of left joins is 1, the number of right joins is 1, the number of full joins is 1, the number of cross joins is 1, the number of hash-type multi-table joins is 3, and the number of nested-type multi-table joins is 1.

[0060] Optionally, the query condition information includes the multi-table join status of the query fields in the SQL query statement and / or the relationship between the column data corresponding to the query fields and the query conditions in the SQL query statement. The multi-table join status is used to indicate whether a multi-table join is performed on the tables corresponding to the query fields. By setting the query condition information to include the multi-table join status of the query fields and / or the relationship between the column data corresponding to the query fields and the query conditions, the correlation between the query condition information and the execution time of the SQL query statement is improved, thereby helping to improve the accuracy of the predicted first and second execution times.

[0061] Optionally, the query field information includes at least one of the following: the number of rows, average value, number of distinct values, number of null values, maximum value, and minimum value of the column data corresponding to the query field in the SQL query statement. By specifically including the aforementioned information in the query field information, the correlation between the query field information and the execution time of the SQL query statement is improved, thereby helping to improve the accuracy of the predicted first and second execution times.

[0062] In one implementation, as shown in Table 1, the number of rows of the column data corresponding to the query field is 7, the average value is 17, the number of different values ​​is 5, the number of null values ​​is 1, the maximum value is 22, and the minimum value is 15.

[0063] Table 1

[0064] age 15 16 15 17 18 null 22

[0065] S202: Predict the first duration and the second duration based at least on execution information.

[0066] The first duration is the execution time of the SQL query when dynamic filtering is applied, and the second duration is the execution time of the SQL query when dynamic filtering is not applied.

[0067] In one implementation, a first weight is configured for the multi-table join information, query condition information, and query field information, respectively. This first weight characterizes the weight of the multi-table join information, query condition information, and query field information when used to predict the first duration. For example, the first weight for the multi-table join information is A, the first weight for the query condition information is B, and the first weight for the query field information is C. Further, after receiving the SQL query statement and obtaining its execution information, the first duration is predicted based on the multi-table join information, query condition information, query field information, and their respective first weights.

[0068] In another implementation, a first execution time is predicted based on a first neural network model and execution information. For example, the feature vector of the execution information is input into the first neural network model to obtain the first execution time output by the first neural network. Here, the first neural network model is a pre-trained neural network model used to predict the execution time of an SQL query statement represented by a feature vector during dynamic filtering.

[0069] In another implementation, a first duration is predicted based on a first duration prediction model and execution information. For example, the feature vector of the execution information is input into the first duration prediction model to obtain the first duration output by the first duration prediction model. The first duration prediction model is a pre-trained multiple linear regression model used to predict the execution duration of an SQL query statement represented by a feature vector during dynamic filtering.

[0070] It should be noted that the prediction principle for the second duration is the same as that for the first duration. The prediction method for the second duration can be referred to the prediction method for the first duration, and will not be repeated here.

[0071] S203: If the first duration is less than or equal to the second duration, perform dynamic filtering on the SQL query statement, that is, filter the data of the large table that performs a multi-table join in at least two tables queried by the SQL query statement.

[0072] Optionally, if the first duration is longer than the second duration, the SQL query statement does not perform dynamic filtering, that is, it does not filter the data of the large table in which a multi-table join is performed in at least two tables queried by the SQL query statement.

[0073] It should be noted that "large table" refers to the table with the larger data volume between two tables. Furthermore, if the data volume in two tables is equal, the "large table" can be either of the two tables. Here, "data volume" can refer to the number of rows.

[0074] In one implementation, at least two tables are included, namely a first table and a second table, which require a multi-table join operation. The first table is the smaller table, and the second table is the larger table. Dynamic filtering is performed on the SQL query statement; that is, before performing the multi-table join operation, filter conditions are generated based on the first table, and the data in the second table is dynamically filtered. It should be noted that the first table and the second table are any two of the at least two tables that require a multi-table join operation.

[0075] In another implementation, at least two tables include a third, a fourth, and a fifth table, which require a multi-table join operation. The third table is the smaller table, the fourth table is the middle table, and the fifth table is the larger table. Dynamic filtering is performed on the SQL query statement; that is, before performing the multi-table join operation, filter conditions are generated based on the third table, and the data in the fifth table is dynamically filtered. It should be noted that the third, fourth, and fifth tables are any three of the at least two tables that require a multi-table join.

[0076] In another implementation, the SQL query queries at least three tables. Taking three tables as an example, let's say the third, fourth, and fifth tables. When the SQL parser generates the execution plan, it converts this into a two-table join. For instance, it executes a join between the third and fourth tables to generate a temporary table `tmp`, and then joins `tmp` with the fifth table, thus transforming the three-table join into a two-table join. Furthermore, when the SQL query queries more than three tables, such as four tables, the principle is the same as for querying three tables; the solution for querying three tables can be referenced, and will not be elaborated further here.

[0077] In the above embodiments, for each received SQL query statement, a first duration and a second duration are determined. Dynamic filtering is performed on the SQL query statement when the first duration is less than or equal to the second duration. On one hand, this achieves automatic determination of whether to perform dynamic filtering on the SQL query statement, realizing adaptive dynamic filtering and reducing the manual cost of performing dynamic filtering, thus solving the problem of high requirements for maintenance personnel in related technologies. On the other hand, since dynamic filtering is only performed on the SQL query statement when the first duration is less than or equal to the second duration, and not performed when the first duration is greater than the second duration, the solution with the shortest execution time is selected. Therefore, the optimal solution is chosen in the determination of whether to perform dynamic filtering, thus avoiding the problem of dynamic filtering causing a decrease in query performance when the time saved by performing dynamic filtering is insufficient to compensate for the time spent generating the filtering conditions. Furthermore, since there is no need to set a preset duration, it also solves the problem of "negative acceleration" that is easily caused in related technologies.

[0078] Optionally, S202 includes: predicting a first duration and a second duration based on execution information and configuration information of the computer device.

[0079] The configuration information includes at least one of the following: hardware environment information of the computer equipment, software environment information of the computer equipment, and current operating environment information of the computer equipment.

[0080] In one implementation, the hardware environment information of the computer device includes at least one of the following: processor model, memory model, motherboard model, and hard disk model.

[0081] In one implementation, the software environment information of the computer device includes at least one of: system version number, software version number, and database engine version number. The software version number is the version number of the software running the SQL query statement.

[0082] In one implementation, the current operating environment information of the computer device includes: processor utilization, memory utilization, and number of nodes. A node represents a computer device executing an SQL query statement.

[0083] In the above embodiments, the first duration and the second duration are predicted based on execution information and computer device configuration information. Based on the execution information, the impact of computer device on the execution time of SQL query statements is comprehensively considered, which helps to improve the accuracy of the predicted first duration and the second duration.

[0084] Optionally, combined Figure 2 ,like Figure 3 As shown, S202 includes:

[0085] S202A: Determine the target feature vector, which is used to represent the execution information.

[0086] Optionally, S202A includes: determining a first feature vector of execution information; and performing dimensionality reduction processing on the first feature vector to obtain a target feature vector. Obtaining the target feature vector by performing dimensionality reduction processing on the first feature vector, and predicting the first duration and the second duration based on the target feature vector, helps to reduce computational complexity.

[0087] In one implementation, Principal Component Analysis (PCA) can be used to reduce the dimensionality of the first eigenvector to obtain the target eigenvector. Of course, other methods can also be used to reduce the dimensionality of the first eigenvector, such as Linear Discriminant Analysis (LDA), Singular Value Decomposition (SVD), and Gaussian Discriminant Analysis (GDA).

[0088] Optionally, when the execution information includes multi-table join information, query condition information, and query field information, determining the first feature vector of the execution information includes: determining the second feature vector of the multi-table join information; determining the third feature vector of the query condition information; determining the fourth feature vector of the query field information; and concatenating the second, third, and fourth feature vectors to obtain the first feature vector.

[0089] In one implementation, determining the second feature vector of multi-table join information includes: traversing each table in the database and determining the identifier of each table; obtaining an array of join information for each table based on an SQL query statement; and obtaining the second feature vector of multi-table join information based on the identifier of each table and the array of join information for each table. The join information for each table includes at least one of the following: the total number of multi-table joins, the number of inner joins, the number of left joins, the number of right joins, the number of full joins, the number of cross joins, the number of hash-type multi-table joins, the number of merge-type multi-table joins, or the number of nested multi-table joins.

[0090] It should be noted that the number of bits in the array of join information for each table is the same as the number of elements in the join information for each table. For example, if the join information for each table only includes the total number of multi-table joins for each table (i.e., the number of elements is 1), the array of join information for each table is a 1-bit array. If the join information for each table includes the total number of multi-table joins for each table, the number of inner joins, and the number of left joins (i.e., the number of elements is 3), the array of join information for each table is a 3-bit array.

[0091] It should be noted that the dimension of the second feature vector is determined based on the maximum number of tables that can be stored in the database and the number of bits in the array of join information for each table.

[0092] In one implementation, determining a second feature vector based on the identifier of each piece of information in the multi-table join information includes: obtaining a multi-table join array based on the identifier of each piece of information; and concatenating the multi-table join array to obtain the second feature vector.

[0093] In one example, assuming a database with a maximum of 500 tables, and each table's join information represented by a 9-bit array, the second feature vector has 500 * 9 = 4500 dimensions. Initially, all values ​​in the second feature vector are 0. Each table is assigned an identifier (id) starting with 0, such as 0 for table A, 1 for table B, and 2 for table C. The join information for table A obtained through an SQL query can be represented as [5, 1, 3, 2, 2, 2, 2, 1, 3]. The join information for table C... The information can be represented as [6, 2, 4, 2, 1, 2, 2, 1, 2]. Then, place the connection information of table A [5, 1, 3, 2, 2, 2, 2, 1, 3] in positions 0*9 to 0*9+(9-1) of the 4500-dimensional vector, and place the connection information of table C [6, 2, 4, 2, 1, 2, 2, 1, 2] in positions 2*9 to 2*9+(9-1) of the 4500-dimensional vector. By putting the connection information of all the tables into the 4500-dimensional feature vector, the determination of the second feature vector is completed.

[0094] In one implementation, determining the third feature vector of query condition information includes: determining the condition feature vector of each query condition; obtaining n condition arrays based on the same dimension of each condition feature vector; and constructing the third feature vector based on the array information of each condition array.

[0095] Where n is the dimension of the conditional feature vector. The array information includes at least one of the following: length, mean, variance, first-order autocorrelation (acf1), number of intersections between the curve and the mean line, trend strength, linearity of the trend, and curvature of the trend.

[0096] In one implementation, determining the condition feature vector for each query condition includes: determining the multi-table join state of the query field corresponding to each query condition, and the size relationship between the column data corresponding to the query field and the query condition; and determining the condition feature vector for each query condition based on the multi-table join state of each query field and the size relationship between the column data corresponding to each query field and the query condition.

[0097] In one example, the third feature vector is represented using four dimensions, and the dimensions of the third feature vector are determined as follows:

[0098] The first dimension is determined based on the multi-table join status of the query fields corresponding to the query conditions. Specifically, when a multi-table join is performed on the table corresponding to the query field, the value of the first dimension is 1, and when no multi-table join operation is performed on the table corresponding to the query field, the value of the first dimension is 0.

[0099] The second dimension is based on the size relationship between the column data corresponding to each query field and the query condition. Specifically, if V >= Vmax, the value of the second dimension is 1; if V < Vmin, the value of the second dimension is 0; otherwise, the value of the second dimension is (V - Vmin) / (Vmax - Vmin), where V is the query condition, Vmax is the maximum value of the field, and Vmin is the minimum value of the field.

[0100] The third dimension is based on the size relationship between the column data corresponding to each query field and the query condition. Specifically, if V <= Vmin, the value of the third dimension is 1; if V > Vmax, the value of the third dimension is 0; otherwise, the value of the third dimension is 1 - (V - Vmin) / (Vmax - Vmin).

[0101] The fourth dimension is based on the size relationship between the column data corresponding to each query field and the query condition. Specifically, if the condition value is greater than the maximum value or the minimum value, the value of the fourth dimension is -1; otherwise, the value of the fourth dimension is (V - Vmin) / (Vmax - Vmin) + 1.

[0102] In one implementation, determining the fourth feature vector of the query field information includes: determining the number of rows, average value, number of different values, number of null values, maximum value, and minimum value of the column data corresponding to the query field in the SQL query statement; and determining the fourth feature vector based on the number of rows, average value, number of different values, number of null values, maximum value, and minimum value.

[0103] In an example, as shown in Table 1, the number of rows of the column data corresponding to the query field is 7, the average value is 17, the number of different values is 5, the number of null values is 1, the maximum value is 22, and the minimum value is 15. The fourth feature vector is (7, 17, 5, 1, 22, 15).

[0104] It should be noted that when the execution information only includes multi-table join information, determining the first feature vector of the execution information includes determining the second feature vector of the multi-table join information. Furthermore, when the execution information includes both multi-table join information and query condition information, determining the first feature vector of the execution information includes: determining the second feature vector of the multi-table join information; determining the third feature vector of the query condition information; and concatenating the second and third feature vectors to obtain the first feature vector. It is understandable that when the execution information only includes query condition information or query field information, the principle for determining the first feature vector of the execution information is the same as the principle for determining the first feature vector when the execution information only includes multi-table join information, and will not be repeated here. Furthermore, when the execution information includes both multi-table join information and query field information, or includes both query condition information and query field information, the principle for determining the first feature vector is the same as the principle for determining the first feature vector when the execution information includes both multi-table join information and query condition information, and will not be repeated here.

[0105] S202B: Predicts the first duration based on the target feature vector and the first duration prediction model.

[0106] The first duration prediction model is used to predict the execution duration of dynamic filtering of an SQL query statement represented by a feature vector.

[0107] In one implementation, such as Figure 4 As shown, the target feature vector is input into the first duration prediction model to obtain the first duration output by the first duration prediction model.

[0108] S202C: Predicts the second duration based on the target feature vector and the second duration prediction model.

[0109] The second duration prediction model is used to predict the execution time of an SQL query statement represented by a feature vector when dynamic filtering is not performed.

[0110] In one implementation, such as Figure 4 As shown, the target feature vector is input into the second duration prediction model to obtain the second duration output by the second duration model.

[0111] In one implementation, such as Figure 5 As shown, the multiple linear regression model is first trained based on historical SQL query statements to obtain a first duration prediction model and a second duration prediction model. Then, based on the first duration prediction model and the second duration prediction model, the first duration and the second duration are predicted respectively. Based on the first duration and the second duration, it is determined whether to perform dynamic filtering on the SQL query statement, and then the SQL query statement is executed based on the determination result.

[0112] It should be noted that when predicting the first and second durations based on execution information and computer device configuration information, the same first and second duration prediction models can be used. For example, one could first determine the first feature vector of the execution information and the fifth feature vector of the configuration information, then concatenate the first and fifth feature vectors to obtain the sixth feature vector. Then, the first duration is obtained based on the sixth feature vector and the first duration prediction model, and the second duration is obtained based on the sixth feature vector and the second duration prediction model. Alternatively, after obtaining the sixth feature vector, one could perform dimensionality reduction on it to obtain the seventh feature vector, then obtain the first duration based on the seventh feature vector and the first duration prediction model, and the second duration is obtained based on the seventh feature vector and the second duration prediction model.

[0113] In the above embodiments, the first duration is predicted by the first duration prediction model and the second duration is predicted by the second duration prediction model. This method is simple, convenient, and efficient, and helps to improve the accuracy of the predicted first and second durations.

[0114] Optionally, such as Figure 6 As shown, the data processing method also includes:

[0115] S204: Obtain training samples, which include historical execution information of historical SQL query statements.

[0116] Among them, the historical SQL query statements are used for at least two sample tables in the database, and the historical execution information includes at least one of the following: historical multi-table join information of at least two sample tables, historical query condition information of historical SQL query statements, and historical query field information of historical SQL query statements.

[0117] In one implementation, such as Figure 7 As shown, historical SQL query statements are obtained, and based on these statements, metadata information, SQL execution plans, and SQL execution log information are determined. Further, training samples are determined based on the metadata information and SQL execution plans. Specifically, historical query field information is determined based on the metadata information, and historical multi-table join information and historical query condition information are determined based on the SQL execution plans. Sample labels and sample durations are determined based on the SQL execution log information. The sample labels include a first sample label and a second sample label. The first sample label represents that dynamic filtering was performed on the historical SQL query statements, and the second sample label represents that dynamic filtering was not performed on the historical SQL query statements. The sample duration includes both the first and second sample durations.

[0118] S205: Determine the sample feature vector, which is used to characterize historical execution information.

[0119] It should be noted that the principle of determining the sample feature vector is the same as that of determining the target feature vector. Therefore, the method for determining the sample feature vector can refer to the method for determining the target feature vector, which will not be repeated here.

[0120] In one implementation, such as Figure 7 As shown, data processing is performed on metadata information to obtain feature vectors for historical query field information, and data processing is performed on SQL execution plans to obtain feature vectors for historical multi-table join information and historical query condition information. Further, by concatenating the feature vectors of historical multi-table join information, historical query condition information, and historical query field information, the feature vector of historical execution information is obtained. Further, the feature vector of historical execution information is subjected to dimensionality reduction processing to obtain sample feature vectors.

[0121] It needs to be explained that, Figure 7 The feature vectors of other information in the target computer device can be feature vectors of the target configuration information of the target computer device. Furthermore, the target computer device can be a computer device used to predict the first duration and the second duration, or it can be other computer devices; this application does not impose any limitations on this.

[0122] S206: Based on the sample feature vector and the duration of the first sample, train a linear regression model to obtain the first duration prediction model.

[0123] The first sample duration is the execution time of the historical SQL query statement when dynamic filtering is performed on the historical SQL query statement.

[0124] Optionally, a linear regression model is trained based on the sample feature vector and the duration of the first sample, including: training a linear regression model based on the sample feature vector, the label of the first sample, and the duration of the first sample.

[0125] In one implementation, the linear regression model is a multiple linear regression model. For example, the multiple linear regression model could be y = β0 + β1x1 + ... + β p x p +∈.

[0126] In one implementation, such as Figure 8 As shown, the duration of the first sample is used as the y-value, and the sample feature vector is used as the x-value to train the multiple linear regression model, determining β0, β1, ..., β2. p , and the first value of ∈, and based on β0, β1, ..., β p , and the first value of ∈, to obtain the first duration prediction model.

[0127] S207: Based on the sample feature vector and the second sample duration, train a linear regression model to obtain the second duration prediction model.

[0128] The second sample duration is the execution time of historical SQL query statements when no dynamic filtering is performed on the historical SQL query statements.

[0129] Optionally, a linear regression model is trained based on the sample feature vector and the second sample duration, including: training a linear regression model based on the sample feature vector, the second sample label, and the second sample duration.

[0130] In one implementation, the linear regression model is a multiple linear regression model. For example, the multiple linear regression model could be y = β0 + β1x1 + ... + β p x p +∈.

[0131] In one implementation, such as Figure 8 As shown, the duration of the second sample is used as the y-value, and the sample feature vector is used as the x-value to train the multiple linear regression model, determining β0, β1, ..., β2. p , and the second value of ∈, and based on β0, β1, ..., β p And the second value of ∈, to obtain the second duration prediction model.

[0132] In the above embodiments, a linear regression model is trained using historical execution information of historical SQL query statements, the duration of the first sample, and the duration of the second sample to obtain a first duration prediction model and a second duration prediction model. This strengthens the correlation between the prediction model and the execution duration of historical SQL query statements, as well as the correlation with the execution duration of dynamic filtering, thereby helping to improve the accuracy of the first duration prediction model and the second duration prediction model in predicting the first duration and the second duration.

[0133] Optionally, the training samples may also include target configuration information of the target computer device. The target computer device is the computer device used to train the linear regression model, and the target configuration information includes at least one of the following: hardware environment information of the target computer device, software environment information of the target computer device, and runtime environment information of the target computer device when running historical SQL queries.

[0134] Optionally, the data processing method further includes: determining the feature vector of the target configuration information; concatenating the sample feature vector and the feature vector of the target configuration information to obtain the target sample feature vector; training a linear regression model based on the target sample feature vector and the first sample duration to obtain a first duration prediction model; and training a linear regression model based on the target sample feature vector and the second sample duration to obtain a second duration prediction model. Since different computer devices have different execution performance, training the linear regression model using the feature vector of the target configuration information and the sample feature vector of the target computer device helps to further improve the accuracy of the duration prediction model in predicting the first and second durations.

[0135] The foregoing primarily describes the solutions of the embodiments of this application from a methodological perspective. It is understood that, in order to achieve the above-described functions, the computer device includes at least one of the hardware structures and software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0136] This application embodiment can divide the data processing device into functional units according to the above method example. For example, each function can be divided into its own functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0137] For example, Figure 9 A possible structural diagram of the data processing apparatus (denoted as data processing apparatus 900) involved in the above embodiments is shown. The data processing apparatus 900 includes an acquisition unit 901, a prediction unit 902, and an execution unit 903. The acquisition unit 901 is used to receive a Structured Query Language (SQL) query statement and determine the execution information of the SQL query statement; the SQL query statement is used to query at least two tables in a database; the execution information includes at least one of the following: multi-table join information of at least two tables, query condition information of the SQL query statement, and query field information of the SQL query statement. For example, Figure 2The step S201 is shown. A prediction unit is used to predict a first duration and a second duration based at least on execution information; wherein the first duration is the execution duration of the SQL query statement when dynamic filtering is performed on the SQL query statement, and the second duration is the execution duration of the SQL query statement when dynamic filtering is not performed on the SQL query statement. For example, Figure 2 The steps S202 shown, and Figure 3 The S202A-S202C shown. Execution unit 903 is used to perform dynamic filtering on SQL query statements when the first duration is less than or equal to the second duration.

[0138] Optionally, the multi-table join information includes at least one of the following: the total number of multi-table joins in at least two tables, the number of inner joins, the number of left joins, the number of right joins, the number of full joins, the number of cross joins, the number of hash-type multi-table joins, the number of merge-type multi-table joins, and the number of nested multi-table joins; and / or the query condition information includes the multi-table join status of the query fields in the SQL query statement and / or the size relationship between the column data corresponding to the query fields and the query filter conditions in the SQL query statement. The multi-table join status is used to characterize whether the tables corresponding to the query fields are performing multi-table joins; and / or the query field information includes at least one of the following: the number of rows, average value, number of distinct values, number of null values, maximum value, and minimum value of the column data corresponding to the query fields in the SQL query statement.

[0139] Optionally, the execution unit is specifically used to: predict a first duration and a second duration based on execution information and configuration information of the computer device; the configuration information includes at least one of the following: hardware environment information of the computer device, software environment information of the computer device, and current operating environment information of the computer device;

[0140] Optionally, the execution unit is specifically used to: determine a target feature vector, which is used to represent execution information; predict a first duration based on the target feature vector and a first duration prediction model; the first duration prediction model is used to predict the execution duration of the SQL query statement represented by a feature vector when performing dynamic filtering; predict a second duration based on the target feature vector and a second duration prediction model; the second duration prediction model is used to predict the execution duration of the SQL query statement represented by a feature vector when dynamic filtering is not performed.

[0141] Optionally, the device further includes a training module for: acquiring training samples, the training samples including historical execution information of historical SQL query statements; the historical SQL query statements are used to query at least two sample tables in the database, and the historical execution information includes at least one of the following: historical multi-table join information of at least two sample tables, historical query condition information of historical SQL query statements, and historical query field information of historical SQL query statements; determining sample feature vectors, the sample feature vectors being used to characterize the historical execution information; training a linear regression model based on the sample feature vectors and a first sample duration to obtain a first duration prediction model; the first sample duration is the execution duration of historical SQL query statements when dynamic filtering is performed on the historical SQL query statements; training a linear regression model based on the sample feature vectors and a second sample duration to obtain a second duration prediction model; the second sample duration is the execution duration of historical SQL query statements when dynamic filtering is not performed on the historical SQL query statements.

[0142] For a detailed description of the above-mentioned optional methods, please refer to the foregoing method embodiments, which will not be repeated here. Furthermore, the explanation of any of the data processing apparatus 900 provided above, as well as the description of its beneficial effects, can be found in the corresponding method embodiments described above, and will not be repeated here.

[0143] As an example, combined Figure 1 The functions implemented by some or all of the acquisition unit 901, prediction unit 902, and execution unit 903 in the data processing device 900 can be achieved through... Figure 1 Processor 101 in the middle executes Figure 1 The program code in memory 102 is used for implementation. The acquisition unit 901 can also be implemented through... Figure 1 The receiving unit in the communication interface 103 is implemented.

[0144] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods executed by any of the computer devices described above.

[0145] For explanations of the relevant content and descriptions of the beneficial effects in any of the computer-readable storage media provided above, please refer to the corresponding embodiments described above, which will not be repeated here.

[0146] This application also provides a chip. This chip integrates a control circuit for implementing the functions of the aforementioned computer device 100 and one or more ports. Optionally, the functions supported by this chip can be referred to above, and will not be repeated here. Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. The aforementioned storage medium can be a read-only memory, random access memory, etc. The aforementioned processing unit or processor can be a central processing unit, a general-purpose processor, an application-specific integrated circuit (ASIC), a microprocessor (digital signal processor, DSP), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0147] This application also provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform any of the methods described in the above embodiments. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or may include one or more data storage devices such as servers or data centers that can be integrated with the medium. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD), etc.

[0148] It should be noted that the devices for storing computer instructions or computer programs provided in the embodiments of this application, such as but not limited to the memory, computer-readable storage medium and communication chip, are all non-transitory.

[0149] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks, SSDs).

[0150] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, the disclosure, and the appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0151] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A data processing method, characterized in that, For use in a computer device, the method includes: The system receives a Structured Query Language (SQL) query statement and determines the execution information of the SQL query statement. The SQL query statement is used to query at least two tables in the database, and the execution information includes multi-table join information of the at least two tables, query condition information of the SQL query statement, and query field information of the SQL query statement. Based on the identifier of each piece of information in the multi-table join information, a multi-table join array is obtained; the multi-table join array is concatenated to obtain the second feature vector of the multi-table join information; Determine the multi-table join status of the query fields corresponding to each query condition, and the size relationship between the column data corresponding to the query fields and the query conditions; based on the multi-table join status of each query field and the size relationship between the column data corresponding to each query field and the query conditions, determine the condition feature vector of each query condition; based on the same dimension of each condition feature vector, obtain n condition arrays; based on the array information of each condition array, construct a third feature vector; the array information includes at least one of the following: length of the condition array, mean, variance, first-order autocorrelation, number of intersections between the curve and the mean line, strength of the trend term, linearity of the trend term, and curvature of the trend term; Determine the fourth feature vector of the query field information; the query field information includes at least one of the following: the number of rows, average value, number of distinct values, number of null values, maximum value, and minimum value of the column data corresponding to the query field in the SQL query statement; The first feature vector is obtained by concatenating the second feature vector, the third feature vector, and the fourth feature vector; The first feature vector is reduced in dimensionality to obtain the target feature vector. The first duration and the second duration are predicted based on the target feature vector; wherein, the first duration is the execution duration of the SQL query statement when dynamic filtering is performed on the SQL query statement, and the second duration is the execution duration of the SQL query statement when the dynamic filtering is not performed on the SQL query statement; If the first duration is less than or equal to the second duration, the dynamic filtering is performed on the SQL query statement.

2. The method according to claim 1, characterized in that, The multi-table join information includes at least one of the following: the total number of multi-table joins in the at least two tables, the number of inner joins, the number of left joins, the number of right joins, the number of full joins, the number of cross joins, the number of hash-type multi-table joins, the number of merge-type multi-table joins, and the number of nested multi-table joins; and / or The query condition information includes the multi-table join status of the query field in the SQL query statement and / or the size relationship between the column data corresponding to the query field and the query condition in the SQL query statement. The multi-table join status is used to indicate whether the table corresponding to the query field has performed a multi-table join.

3. The method according to claim 1, characterized in that, Based on the execution information, predicting the first duration and the second duration includes: Based on the execution information and the configuration information of the computer device, the first duration and the second duration are predicted; the configuration information includes at least one of the hardware environment information of the computer device, the software environment information of the computer device, and the current operating environment information of the computer device.

4. The method according to any one of claims 1-3, characterized in that, Based on the execution information, predicting the first duration and the second duration includes: Based on the target feature vector and the first duration prediction model, the first duration is predicted; the first duration prediction model is used to predict the execution duration of the SQL query statement represented by a feature vector for dynamic filtering. Based on the target feature vector and the second duration prediction model, the second duration is predicted; the second duration prediction model is used to predict the execution duration of an SQL query statement represented by a feature vector when dynamic filtering is not performed.

5. The method according to claim 4, characterized in that, The method further includes: Obtain training samples, which include historical execution information of historical SQL query statements; the historical SQL query statements are used to query at least two sample tables in the database, and the historical execution information includes historical multi-table join information of the at least two sample tables, historical query condition information of the historical SQL query statements, and historical query field information of the historical SQL query statements; Determine a sample feature vector, which is used to characterize the historical execution information; A linear regression model is trained based on the sample feature vector and the first sample duration to obtain the first duration prediction model; the first sample duration is the execution duration of the historical SQL query statement when dynamic filtering is performed on the historical SQL query statement; The linear regression model is trained based on the sample feature vector and the second sample duration to obtain the second duration prediction model; the second sample duration is the execution duration of the historical SQL query statement when no dynamic filtering is performed on the historical SQL query statement.

6. A data processing apparatus, characterized in that, include: The acquisition unit is used to receive a Structured Query Language (SQL) query statement and determine the execution information of the SQL query statement; The SQL query statement is used to query at least two tables in the database; the execution information includes multi-table join information of the at least two tables, query condition information of the SQL query statement, and query field information of the SQL query statement; An execution unit is used to obtain a multi-table join array based on the identifier of each piece of information in the multi-table join information; By concatenating the multi-table join array, a second feature vector of the multi-table join information is obtained; The process involves: determining the multi-table join status of the query fields corresponding to each query condition, and the relationship between the column data corresponding to the query fields and the query conditions; determining the condition feature vector for each query condition based on the multi-table join status of each query field and the relationship between the column data corresponding to each query field and the query conditions; obtaining n condition arrays based on the same dimension of each condition feature vector; constructing a third feature vector based on the array information of each condition array; the array information includes at least one of the following: length of the condition array, average value, variance, first-order autocorrelation, number of intersections between the curve and the mean line, strength of the trend term, linearity of the trend term, and curvature of the trend term; determining a fourth feature vector for the query field information; the query field information includes at least one of the following: number of rows, average value, number of distinct values, number of null values, maximum value, and minimum value of the column data corresponding to the query fields in the SQL query statement; concatenating the second feature vector, the third feature vector, and the fourth feature vector to obtain a first feature vector; and performing dimensionality reduction on the first feature vector to obtain the target feature vector. The prediction unit is used to predict a first duration and a second duration based on the target feature vector; wherein the first duration is the execution duration of the SQL query statement when dynamic filtering is performed on the SQL query statement, and the second duration is the execution duration of the SQL query statement when the dynamic filtering is not performed on the SQL query statement; The execution unit is further configured to perform the dynamic filtering on the SQL query statement when the first duration is less than or equal to the second duration.

7. The apparatus according to claim 6, characterized in that, The multi-table join information includes at least one of the following: the total number of multi-table joins in the at least two tables, the number of inner joins, the number of left joins, the number of right joins, the number of full joins, the number of cross joins, the number of hash-type multi-table joins, the number of merge-type multi-table joins, and the number of nested multi-table joins; and / or The query condition information includes the multi-table join status of the query field in the SQL query statement and / or the size relationship between the column data corresponding to the query field and the query filter conditions in the SQL query statement. The multi-table join status is used to indicate whether the table corresponding to the query field performs a multi-table join. The execution unit is specifically used to: predict the first duration and the second duration based on the execution information and the configuration information of the computer device; the configuration information includes at least one of the hardware environment information of the computer device, the software environment information of the computer device, and the current operating environment information of the computer device; The execution unit is specifically used for: predicting the first duration based on the target feature vector and the first duration prediction model; the first duration prediction model is used to predict the execution duration of the SQL query statement represented by a feature vector when dynamic filtering is performed; predicting the second duration based on the target feature vector and the second duration prediction model; the second duration prediction model is used to predict the execution duration of the SQL query statement represented by a feature vector when dynamic filtering is not performed.

8. The apparatus according to claim 6 or 7, characterized in that, The device further includes a training module for: Obtain training samples, which include historical execution information of historical SQL query statements; the historical SQL query statements are used to query at least two sample tables in the database, and the historical execution information includes historical multi-table join information of the at least two sample tables, historical query condition information of the historical SQL query statements, and historical query field information of the historical SQL query statements; Determine a sample feature vector, which is used to characterize the historical execution information; Based on the sample feature vector and the first sample duration, a linear regression model is trained to obtain the first duration prediction model; the first sample duration is the execution duration of the historical SQL query statement when dynamic filtering is performed on the historical SQL query statement; Based on the sample feature vector and the second sample duration, the linear regression model is trained to obtain the second duration prediction model; the second sample duration is the execution duration of the historical SQL query statement when no dynamic filtering is performed on the historical SQL query statement.

9. A computer device, characterized in that, include: processor; The processor is connected to a memory for storing computer execution instructions, and the processor executes the computer execution instructions stored in the memory to enable the computer device to implement the method as described in any one of claims 1-5.

10. A computer-readable storage medium, characterized in that, Used to store computer instructions that, when executed on a computer, cause the computer to perform the method of any one of claims 1-5.

Images