Model training method and apparatus, device, storage medium, and computer program product

Abstract

The application provides a model training method and device, equipment, a storage medium and a computer program product. The method comprises: obtaining a plurality of candidate query statement samples for a database, deduplicating the plurality of candidate query statement samples, and obtaining at least one target query statement sample; for each target query statement sample, generating a plurality of candidate execution plan samples, and deduplicating the plurality of candidate execution plan samples to obtain at least one target execution plan sample; predicting the performance of each target execution plan sample by using an artificial intelligence model to obtain predicted performance data of each target execution plan sample; obtaining actual performance data of each target execution plan sample, and training the artificial intelligence model based on the predicted performance data and the actual performance data.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a model training method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] In the field of database querying, to address the high overhead of actually executing Structured Query Language (SQL) execution plans during training data construction, cost-estimated training data is primarily used. This involves using cost estimates (i.e., performance data estimates) generated by the database optimizer to replace the actual execution results as training data labels, thus avoiding the time-consuming process of full plan enumeration and actual execution of historical SQL queries. By rapidly constructing the training set without actually executing the SQL execution plan, the time and resource overhead of the data acquisition phase is reduced.

[0003] While the relevant technologies can reduce the resource consumption and time cost of actual execution to some extent, they result in a huge amount of training data, which increases the difficulty and cost of data processing and model training, and cannot meet the demand for efficient model training in practical applications. Summary of the Invention

[0004] This application provides a model training method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve model training efficiency.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a model training method, including:

[0007] Obtain multiple candidate query statement samples for the database, and deduplicate the multiple candidate query statement samples to obtain at least one target query statement sample;

[0008] For each target query statement sample, multiple candidate execution plan samples are generated, and the multiple candidate execution plan samples are deduplicated to obtain at least one target execution plan sample;

[0009] The performance of each target execution plan sample is predicted using an artificial intelligence model, thereby obtaining the predicted performance data for each target execution plan sample.

[0010] Obtain the actual performance data for each target execution plan sample, and train the artificial intelligence model based on the predicted performance data and the actual performance data.

[0011] This application provides a model training apparatus, including:

[0012] The first deduplication module is used to obtain multiple candidate query statement samples for the database, and to deduplicatize the multiple candidate query statement samples to obtain at least one target query statement sample.

[0013] The second deduplication module is used to generate multiple candidate execution plan samples for each target query statement sample, and to deduplicatize the multiple candidate execution plan samples to obtain at least one target execution plan sample.

[0014] The prediction module is used to predict the performance of each target execution plan sample using an artificial intelligence model, and obtain the predicted performance data of each target execution plan sample.

[0015] The training module is used to acquire the actual performance data of each target execution plan sample, and to train the artificial intelligence model based on the predicted performance data and the actual performance data.

[0016] This application provides an electronic device for model training, the electronic device comprising:

[0017] Memory is used to store computer programs or computer-executable instructions.

[0018] The processor, when executing a computer program or computer-executable instructions stored in the memory, implements the model training method provided in the embodiments of this application.

[0019] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implement the model training method provided in this application.

[0020] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implements the model training method provided in this application.

[0021] The embodiments of this application have the following beneficial effects:

[0022] By deduplicating candidate query statements, redundant query statements can be effectively removed. Further deduplication of candidate execution plans in the target execution plan sample can effectively remove redundant execution plans. This two-layer deduplication of query statements and execution plans avoids redundant computation caused by the AI ​​model's performance prediction of redundant query statement execution plans and redundant execution plans. This effectively reduces the size of the training data, avoids interference from redundant training samples on model convergence, improves model training efficiency, and saves related computing resources. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the architecture of the model training system provided in the embodiments of this application;

[0024] Figure 2 This is a schematic diagram of the structure of the electronic device 500 for model training provided in an embodiment of this application;

[0025] Figure 3A This is a schematic diagram of the first process of the model training method provided in the embodiments of this application;

[0026] Figure 3B This is a schematic diagram of the second process of the model training method provided in the embodiments of this application;

[0027] Figure 3C This is a schematic diagram of the third process of the model training method provided in the embodiments of this application;

[0028] Figure 3D This is a schematic diagram of the fourth process of the model training method provided in the embodiments of this application;

[0029] Figure 4 This is a schematic diagram illustrating the deduplication of candidate query statement samples and candidate execution plan samples provided in the embodiments of this application;

[0030] Figure 5 This is a schematic diagram of graph construction provided in an embodiment of this application;

[0031] Figure 6 This is a flowchart illustrating the model training method provided in the embodiments of this application;

[0032] Figure 7 This is a flowchart illustrating the model training method provided in the embodiments of this application;

[0033] Figure 8 This is a schematic diagram of two-layer data deduplication based on query fingerprint and execution plan provided in an embodiment of this application;

[0034] Figure 9 This is a schematic diagram of differential training in the incremental training scenario provided in the embodiments of this application.

[0035] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0038] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0041] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0042] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0043] 1) Responding to: used to indicate the conditions or states on which the operation is performed depends. When the conditions or states on which it depends are met, one or more operations can be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0044] 2) Artificial Intelligence Models (AI Models): Also known as large models, these are models with a large number of parameters built from artificial neural networks. They are typically pre-trained on massive datasets using self-supervised or semi-supervised learning, and then further optimized for performance and capabilities through fine-tuning and human alignment. Large models are characterized by a large number of parameters, large training data sets, and significant computational resources. They possess the ability to solve general tasks, follow human instructions, and perform complex reasoning. Major categories of AI models include large language models, large vision models, multimodal large models, and large models for basic science. Currently, large models have been widely applied in various fields, including search engines, intelligent agents, related vertical industries, and basic science, driving the intelligent development of various industries.

[0045] 3) Database Query Optimizer: This is the core module of the database management system, primarily responsible for parsing user-submitted SQL queries. By analyzing the semantics of the query and combining it with database statistics (such as data distribution and index status), the database query optimizer evaluates various possible execution paths and ultimately generates an efficient execution plan to improve database query efficiency and overall system performance.

[0046] 4) Optimizer Cost Estimation: This refers to the quantitative evaluation method used by the database query optimizer when selecting an execution plan. Optimizer cost estimation estimates the resources required to perform different operations (such as full table scans, index lookups, data sorting, etc.), including Central Processing Unit (CPU) time, disk I / O, and memory usage. It assigns a cost value to each execution path, thus providing a basis for selecting the optimal execution plan.

[0047] 5) Structured Query Language Execution Plan (SQL Execution Plan): The execution plan is a set of specific instructions and strategies used by the database to execute SQL queries. It instructs the database to access table data in specific ways, including using specific indexes, employing specific table join methods (such as nested loop joins and hash joins), and specific data filtering and sorting steps. The execution plan directly impacts query execution efficiency.

[0048] 6) Machine Learning Models: These are mathematical models built using machine learning algorithms. They can learn from large amounts of data to uncover patterns, regularities, and features. In the field of database query optimization, machine learning models can learn the data characteristics of historical query statements and their corresponding execution plans, thereby predicting the optimal execution plan for new queries and assisting in database optimization decisions.

[0049] 7) Data redundancy: This refers to duplicate, redundant, or information that can be derived from other data within the data. In training data for database queries, data redundancy manifests as a large number of similar or identical SQL query statements and their corresponding execution plans. Redundant data consumes additional storage resources, increases the time and computational cost of data processing and model training, reduces training efficiency, and affects the model's generalization ability and optimization performance.

[0050] 8) Plan Enumeration: This refers to the database query optimizer exhaustively enumerating a large number of possible execution plans (i.e., execution plans) for a query statement based on database metadata and SQL query statements using a specific algorithm. These plans include table access order, data retrieval methods, table join strategies, etc. The purpose of plan enumeration is to provide candidate execution plans and then select the optimal solution through cost evaluation or other methods to improve query performance.

[0051] 9) Loss value: Used to represent the difference between the model's output and the actual result (or labeled result). Model training mainly involves two processes: forward propagation and back propagation. Taking a neural network model including an input layer, hidden layer, and output layer as an example, forward propagation refers to processing the data sequentially through the input layer, hidden layer, and output layer to obtain the final output result; back propagation refers to propagating the calculated loss value sequentially to the output layer, hidden layer, and input layer, thereby updating the weight parameters in each layer.

[0052] This application provides a model training method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve model training efficiency.

[0053] The model training method provided in this application embodiment can be implemented by the terminal alone; or it can be implemented by the terminal and the server in collaboration. For example, the terminal can undertake the model training method described below alone, or the terminal can send a training request for the artificial intelligence model to the server, and the server can execute the model training method according to the received training request for the artificial intelligence model to train the artificial intelligence model, and perform a performance prediction task based on the trained artificial intelligence model.

[0054] The electronic devices provided in this application can be various types of terminal devices or servers. The terminals can be implemented as laptops, tablets, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices, in-vehicle devices), smartphones, smart speakers, smartwatches, smart TVs, in-vehicle terminals, and other types of user terminals. The servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminals and servers can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0055] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the model training system 10 provided in this application embodiment. The terminal 200 is connected to the server 100 through the network 300, wherein the network 300 can be a wide area network or a local area network, or a combination of the two.

[0056] In some embodiments, taking the electronic device as a terminal as an example, the model training method provided in this application embodiment can be implemented by the terminal. For example, a model training plugin can be embedded in the client running in the terminal 200 to implement the model training method locally on the client. That is, the terminal 200 calls the model training plugin to implement the model training method, which involves obtaining multiple candidate query statement samples for the database, deduplicating the multiple candidate query statement samples to obtain at least one target query statement sample; generating multiple candidate execution plan samples for each target query statement sample, and deduplicating the multiple candidate execution plan samples to obtain at least one target execution plan sample; performing performance prediction on each target execution plan sample using an artificial intelligence model to obtain predicted performance data for each target execution plan sample; obtaining actual performance data for each target execution plan sample, and training the artificial intelligence model based on the predicted performance data and the actual performance data to obtain the trained artificial intelligence model. Terminal 200 obtains the query statement for the database, calls the performance prediction plugin based on the query statement to implement the performance prediction method, generates multiple candidate execution plans for the query statement, performs performance prediction on each candidate execution plan through the trained artificial intelligence model, obtains the predicted performance data of each candidate execution plan, selects the optimal execution plan from the multiple candidate execution plans based on the predicted performance data of each candidate execution plan, and actually executes the optimal execution plan to respond to the query request corresponding to the query statement.

[0057] In some embodiments, the model training method provided in this application can also be implemented collaboratively by a server and a terminal. For example, after the terminal 200 obtains a training request for an artificial intelligence model, it calls the model training interface of the server 100 (which can be provided as a cloud service, i.e., a model training service). The server 100 implements the model training method through a model training plugin, obtains multiple candidate query statement samples for the database, deduplicates the multiple candidate query statement samples to obtain at least one target query statement sample; for each target query statement sample, it generates multiple candidate execution plan samples, and deduplicates the multiple candidate execution plan samples to obtain at least one target execution plan sample; it performs performance prediction on each target execution plan sample using an artificial intelligence model to obtain predicted performance data for each target execution plan sample; it obtains the actual performance data for each target execution plan sample, and trains the artificial intelligence model based on the predicted performance data and the actual performance data to obtain the trained artificial intelligence model. The server 100 returns the trained artificial intelligence model to the terminal 200. Terminal 200 obtains the query statement for the database, calls the performance prediction plugin based on the query statement to implement the performance prediction method, generates multiple candidate execution plans for the query statement, performs performance prediction on each candidate execution plan through the trained artificial intelligence model, obtains the predicted performance data of each candidate execution plan, selects the optimal execution plan from the multiple candidate execution plans based on the predicted performance data of each candidate execution plan, and actually executes the optimal execution plan to respond to the query request corresponding to the query statement.

[0058] It is worth noting that the trained artificial intelligence model can be stored locally on server 100 and retrieved when needed, or it can be returned to terminal 200.

[0059] In some embodiments, the terminal or server can implement the model training method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run, such as search engine applications; or they can be applets embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0060] The structure of the electronic device for model training provided in the embodiments of this application is described below. See also... Figure 2 , Figure 2This is a schematic diagram of the structure of the electronic device 500 for model training provided in an embodiment of this application. The explanation uses an example where the electronic device 500 is a terminal. Figure 2 The illustrated electronic device 500 for model training includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 540.

[0061] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0062] Memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory. Memory 550 may optionally include one or more storage devices physically located away from processor 510.

[0063] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0064] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0065] The network communication module 552 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0066] In some embodiments, the model training apparatus provided in this application can be implemented in software. The model training apparatus provided in this application can be provided in various software embodiments, including various forms such as applications, software, software modules, scripts or code.

[0067] Figure 2 A model training device 555 stored in memory 550 is shown. It can be software in the form of programs and plug-ins, and includes a series of modules, including a first deduplication module 5551, a second deduplication module 5552, a prediction module 5553, and a training module 5554. These modules are logically related and can therefore be arbitrarily combined or further split according to the functions they implement. The functions of each module will be described below.

[0068] As mentioned above, the model training method provided in this application can be implemented by various types of electronic devices, such as terminals, servers, or a combination of both. Therefore, the executing entity of each step will not be described again below. See Figure 3A , Figure 3A This is a flowchart illustrating the model training method provided in the embodiments of this application, combined with... Figure 3A The steps shown are explained.

[0069] In step 101, multiple candidate query statement samples for the database are obtained, and the multiple candidate query statement samples are deduplicated to obtain at least one target query statement sample.

[0070] Among them, the candidate query statement samples for the database include SQL historical query statements. These SQL historical query statements can be directly identified as candidate query statement samples, or they can be preprocessed to obtain candidate query statement samples. For example, preprocessing can involve cleaning the SQL historical query statements to remove empty or invalid query statements; preprocessing can also involve desensitizing sensitive information, such as replacing specific data values ​​with placeholders (e.g., WHERE id=123 → WHERE id=?), processing privacy data such as account information; and preprocessing can also involve special character processing, such as uniform quotation marks, removing redundant spaces or newlines, etc.

[0071] Here, there are a large number of duplicate query statement samples in the candidate query statement sample (i.e., duplicate query statement samples). In this embodiment of the application, a deduplication operation is performed to delete duplicate query statement samples in the candidate query statement sample to obtain the target query statement sample, thereby achieving query statement layer deduplication. This can eliminate redundant query statements in the overall query statement and greatly reduce the unnecessary overhead caused by redundant query statements.

[0072] like Figure 4As shown, taking multiple candidate query statement samples as candidate query statement sample 1, candidate query statement sample 2, and candidate query statement sample 3 as an example, candidate query statement sample 2 and candidate query statement sample 3 are duplicate query statements of candidate query statement sample 1. Therefore, duplicate query statements are removed from candidate query statement sample 1, candidate query statement sample 2, and candidate query statement sample 3 to achieve deduplication, resulting in candidate query statement sample 1, which is then determined as the target query statement sample 1.

[0073] See Figure 3B , Figure 3B This is a flowchart illustrating the model training method provided in an embodiment of this application. Figure 3B Show Figure 3A The step 101, "to deduplicate multiple candidate query statement samples and obtain at least one target query statement sample", can be achieved through steps 1011-1013.

[0074] In step 1011, fingerprint information of each candidate query statement sample is extracted.

[0075] Among them, the fingerprint information of the candidate query statement sample is the identification information that represents the essential characteristics of the candidate query statement sample. By using fingerprint information that can represent the essence of the candidate query statement sample without containing specific data, a comparable feature representation is provided for subsequent clustering analysis.

[0076] In some embodiments, step 1011 can be implemented as follows: semantic extraction is performed on each candidate query statement sample to obtain the semantic information of each candidate query statement sample; the semantic information of each candidate query statement sample is hashed to obtain the fingerprint information of each candidate query statement sample.

[0077] For example, for each candidate query sample, lexical analysis is used to remove specific parameters, retaining only the core semantic information. For instance, for the candidate query samples "SELECT * FROM users WHERE age > 25" and "SELECT * FROM users WHERE age > 30", after parameter removal using lexical analysis, the core semantic information is "select all records from the users table that satisfy age > a certain value". Subsequently, a hash algorithm, such as Message Digest Algorithm 5 (MD5) or Secure Hash Algorithm 256-bit (SHA-256), is used to calculate the semantic information of each candidate query sample, generating a fingerprint (Digest) as an identifier. This fingerprint encompasses the core semantics of the candidate query sample. It should be noted that the embodiments of this application are not limited to the semantic extraction method or the hash algorithm.

[0078] Thus, by combining semantic extraction and hash processing, this embodiment of the application can generate fingerprint information that accurately reflects the core semantic features of candidate query statement samples and is easy to compare efficiently, thereby improving the accuracy and processing efficiency of subsequent clustering.

[0079] In some embodiments, fingerprint information can refer to identifier information extracted after structuring and standardizing candidate query statement samples, which can characterize the essential features of the query. Step 1011 can be implemented in at least one of the following ways: 1) Standardizing the candidate query statement samples, such as unifying keyword case, parameterizing specific values ​​as placeholders, and removing redundant spaces and comments; 2) Extracting the syntactic structure features of the candidate query statement samples, such as query type (SELECT / INSERT, etc.), the combination of table names and column names involved, the clauses used (WHERE / JOIN / GROUP BY, etc.) and their frequency of occurrence; 3) Analyzing the pattern features of the candidate query statement samples, including nesting level, type and number of join operations, type of conditional predicate (equal / range / LIKE, etc.), and sorting and grouping features. The above structured features are converted into numerical vectors or hash values ​​to form fingerprint information that can characterize the essence of the candidate query statement samples without containing specific data, providing a comparable feature representation for subsequent clustering analysis.

[0080] In step 1012, based on the fingerprint information of multiple candidate query statement samples, multiple candidate query statement samples are clustered to obtain multiple clusters, wherein each cluster includes at least one candidate query statement sample.

[0081] In this process, based on the fingerprint information of multiple candidate query statement samples, multiple candidate query statement samples are clustered to obtain multiple clusters. The candidate query statement samples in each cluster have similar or even the same semantic information.

[0082] As an example, the specific process of clustering based on the fingerprint information of candidate query statement samples includes: First, the extracted fingerprint information (including parameterized query templates, structural feature vectors, etc.) is subjected to feature space construction and dimensionality normalization; then, clustering algorithms suitable for high-dimensional sparse data (such as improved K-Means++, hierarchical clustering algorithms) are used to group candidate query statement samples with similarity in semantics and structure into the same category by calculating the similarity distance between fingerprint information (such as Jaccard similarity, edit distance, or cosine similarity); during the clustering process, a dynamic threshold mechanism can be set, and when the distance between a candidate query statement sample and the cluster center exceeds the threshold, a new cluster is automatically generated; finally, multiple clusters with clear semantic features are output, and the candidate query statement samples in each cluster have highly similar structural patterns and semantic intents, while different clusters maintain obvious differences, providing a structured grouping basis for subsequent sampling.

[0083] In step 1013, at least one target query statement sample is obtained from at least one cluster.

[0084] In this embodiment, candidate query statement samples in each cluster can be sampled, and all candidate query statement samples obtained from the clusters can be determined as target query statement samples; in this embodiment, candidate query statement samples in some clusters can also be sampled, and candidate query statement samples obtained from some clusters can be determined as target query statement samples.

[0085] Thus, the beneficial effects achieved by steps 1011 to 1013 in this embodiment are as follows: by extracting the fingerprint information of candidate query statement samples and performing cluster analysis based on the fingerprint information, it is possible to intelligently identify and classify duplicate query statement samples with similar structures but different surface forms from a semantic level, overcoming the limitations of deduplication methods based on string exact matching; by adopting a post-clustering sampling method, it not only ensures the representativeness of the target query statement samples and covers various query patterns, but also effectively reduces the redundancy of the target query statement samples, providing a high-quality and diverse sample foundation for subsequent query optimization or model training, while significantly improving the efficiency and accuracy of large-scale query processing.

[0086] In some embodiments, step 1013 can be implemented as follows: when the number of candidate query statement samples included in any cluster is greater than or equal to a first quantity threshold, sample target query statement samples from any cluster with a number less than the first quantity threshold; when the number of candidate query statement samples included in any cluster is less than the first quantity threshold, determine the candidate query statement samples included in any cluster as target query statement samples.

[0087] For example, if the number of candidate query statement samples in a certain cluster is large (i.e., the number of candidate query statement samples included in the cluster is greater than or equal to the first number threshold), then the candidate query statement samples in the cluster are randomly sampled so that each cluster retains a set number of candidate query statement samples (e.g., a maximum of 3 to 10 different candidate query statement samples) to obtain the deduplicated candidate query statement samples (i.e., the target query statement samples). This ensures that enough samples are retained for subsequent training and effectively removes redundant data, greatly reducing the unnecessary overhead caused by redundant query statements.

[0088] Thus, by setting a first quantity threshold and employing a differentiated sampling strategy, this embodiment of the application can intelligently balance the representativeness and diversity of the target query statement samples. For high-frequency query patterns (large clusters, i.e., clusters with a quantity greater than or equal to the first quantity threshold), downsampling avoids data redundancy and improves processing efficiency; for low-frequency query patterns (small clusters, i.e., clusters with a quantity less than the first quantity threshold), all samples are retained to ensure the feature integrity of the query statement. This adaptive sampling mechanism optimizes the sample set size while ensuring sufficient coverage of various query patterns, thereby providing a high-quality sample set that is both representative and diverse for subsequent query analysis or model training, effectively improving the accuracy and generalization ability of query optimization or performance prediction.

[0089] In step 102, multiple candidate execution plan samples are generated for each target query statement sample, and the multiple candidate execution plan samples are deduplicated to obtain at least one target execution plan sample.

[0090] After obtaining the target query sample, to acquire diverse execution plan samples for model training, methods such as prompting intervention and on / off adjustments can be used for each target query sample to assist the database optimizer in generating differentiated candidate execution plan samples. These differentiated candidate execution plan samples refer to a series of different execution plans (i.e., candidate execution plan samples) generated by the database optimizer for the same target query sample through prompting intervention and on / off adjustments. These execution plans differ in execution path, resource usage, and operation order, and cover as many possible execution scenarios as possible, thereby increasing the probability of selecting the optimal execution plan.

[0091] Taking hint intervention as an example, hint intervention refers to explicitly adding database-specific hints to the target query statement sample, thereby adjusting the execution plan without changing the semantics of the target query statement sample. These hints are used to give explicit instructions to the database optimizer, instructing it to select indexes, determine the table join order, etc. For example, in MySQL, an index hint like / *+INDEX(table_nameindex_name)* / can be used. When executing a complex query statement, if the database optimizer does not select an appropriate index, it will lead to poor query performance. However, by adding this index hint, the database optimizer can be forced to use the index specified by the index hint, thereby changing the execution plan and generating candidate execution plans. In MySQL, the hint / *+JOIN_PREFIX(t1,t2)* / can be used to instruct the database optimizer to perform join operations according to the order in which the tables appear as indicated in the hint, instead of the database optimizer evaluating the join order itself. This method can precisely intervene in certain key aspects of the execution plan to obtain different execution paths, thus resulting in diversified execution plans.

[0092] Taking the switch adjustment method as an example, switch adjustment refers to adjusting the database optimizer switches. Optimizer switches are a series of pre-set parameters in the database system. By adjusting these parameters, the optimization strategy of the database optimizer can be changed, affecting its preference for operations such as join methods and scan methods. By reasonably adjusting these optimizer switch parameters, the database optimizer can be guided to generate different types of execution plans (i.e., candidate execution plan samples). Taking MySQL as an example, the system variable `optimizer_switch` controls the on / off state of various optimization strategies, dynamically enabling or disabling different optimization strategies of the query optimizer. Specifically, the `index_merge` variable controls whether the index merge optimization strategy is enabled. When `index_merge = on`, the index merge optimization strategy is enabled; when `index_merge = off`, the index merge optimization strategy is disabled. When index merging optimization is enabled, the database optimizer may merge multiple indexes to satisfy query conditions, resulting in a significantly different execution plan compared to when it is disabled. Similarly, the `block_nested_loop` and `batched_key_access` parameters are system variables in MySQL used to optimize table join operations, thus affecting query execution and performance optimization. The `block_nested_loop` parameter controls whether Block Nested Loop Join (BNLJ) is enabled, an algorithm that reduces the number of accesses to the inner table by caching rows from the outer loop. When this parameter is set to ON (the default value), MySQL reads rows from the outer table into memory blocks in batches when processing joins without suitable indexes, and then matches them with the inner table in batches, reducing disk I / O. The `batched_key_access` parameter optimizes index access efficiency. When enabled, MySQL uses a batch key access (BKA) strategy. BKA sends multiple query conditions to the storage engine in batches, leveraging the ordered nature of the index to prefetch data and reduce random I / O. Adjusting these two parameters can change MySQL's strategy for handling table joins, thus affecting the execution method of join operations and the overall execution plan.

[0093] Here, there are a large number of duplicate candidate execution plan samples (i.e., duplicate execution plan samples) in the candidate execution plan sample. In this embodiment of the application, a deduplication operation is performed to delete the duplicate execution plan samples in the candidate execution plan sample to obtain the target execution plan sample, thereby realizing execution layer deduplication. This can eliminate redundant execution plans in the overall execution plan sample and greatly reduce the unnecessary overhead caused by redundant execution plans.

[0094] like Figure 4 As shown, continuing from the previous example, after deduplicating candidate query statement samples 1, 2, and 3 to obtain target query statement sample 1, the database optimizer generates candidate execution plan samples 1, 2, and 3 for target query statement sample 1. Candidate execution plan sample 2 is a duplicate execution plan of candidate execution plan sample 1; therefore, duplicate execution statements are removed from candidate execution plan samples 1, 2, and 3 to achieve deduplication, resulting in candidate execution plan sample 1 and 3. Candidate execution plan sample 1 is then designated as target execution plan sample 1, and candidate execution plan sample 3 is designated as target execution plan sample 2.

[0095] See Figure 3C , Figure 3C This is a flowchart illustrating the model training method provided in an embodiment of this application. Figure 3C Show Figure 3A The step 102, "to deduplicate multiple candidate execution plan samples and obtain at least one target execution plan sample", can be achieved through steps 1021-1022.

[0096] In step 1021, each candidate execution plan sample is hashed to obtain the fingerprint information of each candidate execution plan sample.

[0097] The fingerprint information of the candidate execution plan sample is a unique identifier that characterizes the essential features of the candidate execution plan sample. The fingerprint information of the candidate execution plan sample is the identifier obtained by hashing the candidate execution plan sample using a hash algorithm. However, the embodiments of this application are not limited to the form of the hash algorithm, such as Message Digest Algorithm Version 5 (MD5), Secure Hash Algorithm 256-bit (SHA-256), etc.

[0098] In some embodiments, step 1021 can be implemented as follows: construct a graph for each candidate execution plan sample to obtain a graph structure corresponding to each candidate execution plan sample; perform graph hashing on the graph structure corresponding to each candidate execution plan sample to obtain fingerprint information of each candidate execution plan sample.

[0099] For example, the candidate execution plan samples generated for a target query statement sample may contain a large number of duplicates, causing data redundancy. To solve this problem, this embodiment of the application models the generated candidate execution plan samples as directed acyclic graphs (i.e., graph structures). Then, a unique hash value is calculated for each directed acyclic graph using a graph hashing algorithm, which serves as the fingerprint information (Plan ID) of the candidate execution plan sample. Of course, this embodiment of the application is not limited to directed acyclic graphs; it can also be undirected acyclic graphs, etc.

[0100] Thus, by transforming candidate execution plan samples into graph structures and performing graph hashing, this embodiment of the application can completely preserve the topological structure and operational dependencies of the candidate execution plan samples, thereby generating fingerprint information that can accurately characterize the essential features of the execution plan (such as operator connection order, parallelism distribution, etc.) and facilitate efficient comparison. Compared with linear hashing, this graph structure-based method is better at capturing deep pattern differences in complex execution plans, effectively solving the misjudgment problem caused by operator permutation or local mutation, providing a more accurate comparison benchmark for execution plan deduplication, similarity analysis, and performance prediction, while ensuring computational efficiency during large-scale processing through the fixed-length encoding of graph hashing.

[0101] In some embodiments, graph construction is performed on each candidate execution plan sample to obtain the graph structure corresponding to each candidate execution plan sample. This can be achieved by performing the following processing on each candidate execution plan sample: obtaining at least one execution step of the candidate execution plan sample; determining each execution step as a node, and determining the edges between the nodes corresponding to the execution steps based on the dependencies between the execution steps; and constructing the graph structure corresponding to the candidate execution plan sample based on the edges between the nodes and the nodes.

[0102] For example, during graph construction, each operation step (i.e., execution step) in the execution plan, such as scanning a table, filtering data, joining tables, etc., is abstracted as a node in the graph structure, and the dependencies between operation steps are represented as directed edges in the graph structure.

[0103] like Figure 5As shown, taking a specific execution plan sample as an example, assuming the query involves a join operation between the users table and the orders table, the sample execution plan includes the following steps: First, perform an index scan on the users table, then perform an index scan on the orders table, then merge the two scan results through a hash join, and finally sort and output the join result. When constructing the graph structure, the execution step of "index scan of the users table" is first transformed into a node (node ​​1), marked as an index scan type, and its table name and other attributes are recorded; then, the execution step of "index scan of the orders table" is transformed into another independent node (node ​​2); next, the execution step of "hash join" is transformed into a third node (node ​​3), and connected to the first two nodes by a directed edge, indicating that the join operation depends on the first two scan results; finally, the execution step of "sorting output" is transformed into a fourth node (node ​​4), and connected to the hash join node by an edge, indicating that the input of the sorting operation comes from the join result.

[0104] Thus, by abstracting each execution step in the candidate execution plan sample as a graph node and constructing edge connections based on the dependencies between execution steps, this embodiment of the application can completely preserve the topological structure and operational timing relationships of the candidate execution plan sample, thereby generating a graph structure that accurately represents the characteristics of the execution path. This graph model-based representation method can not only capture complex relationships such as parallel execution and nested sub-plans that are ignored by traditional linear representations, but also effectively identify the essential differences between candidate execution plan samples in dimensions such as operation order and data flow direction through the topological characteristics of the graph structure. This provides a more accurate structured foundation for subsequent graph hashing and plan similarity analysis, significantly improving the accuracy and efficiency of execution plan optimization.

[0105] In step 1022, based on the fingerprint information of multiple candidate execution plan samples, the multiple candidate execution plan samples are deduplicated to obtain at least one target execution plan sample.

[0106] Among them, multiple candidate execution plan samples are for the same target query statement sample. That is, based on the fingerprint information of multiple candidate execution plan samples for each target query statement sample, multiple candidate execution plan samples for each target query statement sample are deduplicated to obtain at least one target execution plan sample for each target query statement sample.

[0107] For example, deduplication based on the fingerprint information of candidate execution plan samples can be performed by keeping only candidate execution plan samples with different fingerprint information, obtaining deduplicated candidate execution plan samples, and then determining the deduplicated candidate execution plan samples as the target execution plan samples. The deduplication operation can eliminate redundant execution plan samples.

[0108] Of course, step 1022 can also be similar to steps 1012 and 1013, that is, based on the fingerprint information of multiple candidate execution plan samples, multiple candidate execution plan samples are clustered to obtain multiple clusters, wherein each cluster includes at least one candidate execution plan sample, and then at least one target execution plan sample is sampled from at least one cluster.

[0109] Thus, the beneficial effects achieved by steps 1021 to 1022 in this embodiment of the application are as follows: by extracting the fingerprint information of candidate execution plan samples and performing deduplication based on the fingerprint information, redundant execution plan samples with the same logical structure can be accurately identified and eliminated, while preserving the diversity of different execution strategies. This deduplication method based on fingerprint information not only solves the problem that parameterized variants cannot be identified based on text comparison, but also efficiently processes a large number of candidate execution plan samples through hash operations, significantly reducing the computational overhead of subsequent optimization analysis, while ensuring that the target execution plan samples are neither redundant nor completely cover various execution modes, providing a high-quality data foundation for model training and execution plan analysis.

[0110] In step 103, the performance of each target execution plan sample is predicted using an artificial intelligence model to obtain the predicted performance data for each target execution plan sample.

[0111] In the field of database query optimization, artificial intelligence models can learn the data characteristics of query statements and their corresponding execution plans, thereby predicting the performance data of the execution plans. Based on the predicted performance data, the optimal execution plan can be selected from multiple target execution plan samples to assist in database optimization decisions. The embodiments of this application are not limited to the model structure of the artificial intelligence model; for example, the artificial intelligence model can be a deep neural network, a machine learning model, a gradient boosting decision tree model, a graph neural network model, etc. Furthermore, the embodiments of this application are not limited to performance data. Performance data can be time-dimensional indicators, including estimated execution time (such as total execution time, number of executions, average runtime, average CPU runtime, and total CPU runtime); performance data can be resource-dimensional indicators, including at least one of CPU usage, memory usage, and disk I / O counts; performance data can be risk-dimensional indicators, including at least one of performance risk level, potential performance bottleneck identification, and abnormal situation warnings.

[0112] In some embodiments, the target execution plan sample is parsed into structured feature data using an artificial intelligence model; the structured feature data is preprocessed; and the preprocessed feature data is used for performance prediction to obtain the predicted performance data of the target execution plan sample.

[0113] For example, the target execution plan sample can be parsed into structured feature data, which includes at least one of the following: operator sequence features, including but not limited to scan operator type, join operator type, aggregation operator type and their order; execution path topology features, including data dependencies between operators and parallel execution branches; resource estimation features, including cost estimates, estimated number of rows and memory usage provided by the optimizer; and contextual features, including database version identifier, hardware configuration parameters and concurrent load indicators.

[0114] For example, preprocessing of structured feature data can be achieved in the following ways: standardizing the numerical features in the structured feature data to make them conform to a normal distribution; embedding vectorization or one-hot encoding of the categorical features in the structured feature data; and extracting topological features of the graph structure features in the structured feature data through graph neural networks to generate graph embedding vectors of fixed dimensions.

[0115] Thus, this application embodiment achieves accurate prediction of execution plan performance through artificial intelligence modeling technology, overcomes the limitations of traditional cost models that rely on static assumptions, provides data-driven decision support for query optimization, and significantly improves the performance optimization effect of database systems.

[0116] In step 104, the actual performance data of each target execution plan sample is obtained, and an artificial intelligence model is trained based on the predicted performance data and the actual performance data.

[0117] In this process, each target execution plan sample is executed in the database to obtain the actual performance data of each target execution plan sample. Of course, the embodiments of this application are not limited to actual performance data. Actual performance data can be time-dimensional indicators, including actual execution time (such as total execution time, number of executions, average runtime, average CPU runtime, and total CPU runtime); actual performance data can be resource-dimensional indicators, including at least one of CPU usage, memory usage, and disk I / O counts; actual performance data can be risk-dimensional indicators, including at least one of performance risk level, potential performance bottleneck identification, and abnormal situation warning.

[0118] For example, based on the actual performance data and predicted performance data of each target execution plan sample, the loss (i.e., loss value) corresponding to each target execution plan sample is determined. The parameters of the artificial intelligence model are then updated based on the loss corresponding to each target execution plan sample, resulting in the trained artificial intelligence model. Of course, the embodiments of this application are not limited to the form of the loss function of the artificial intelligence model. Different loss functions can be selected for training according to actual needs and data characteristics. For example, it can be the mean squared error (MSE) loss function, or it can be the binary cross-entropy (BCE) loss function.

[0119] Taking the Mean Squared Error (MSE) loss function as an example, for a single target execution plan sample, let its actual performance data be y, and its predicted performance data be... Then the loss corresponding to the i-th target execution plan sample is The basic MSE loss corresponding to the artificial intelligence model is Of course, multi-granularity MSE loss can also be considered. Suppose the target execution plan sample contains m operators, and the actual execution time and predicted execution time of the j-th operator are t and t, respectively. j and The operator-level loss is Weights are assigned to resource metrics such as CPU and memory. i (e.g. w) CPU =0.6, w Mem =0.4), then the resource-level loss is Combining the above components, a composite loss function L is constructed. total =αL base +βL op +γL resource , where α+β+γ=1.

[0120] Taking the Binary Cross Entropy (BCE) loss function as an example, this embodiment of the application adopts a pairwise training method, using the BCE loss function for model training. The specific process is as follows:

[0121] Taking four target execution plan samples as an example, these four samples are paired, resulting in six plan pairs according to combinatorial mathematics principles. For each plan pair, a binary label is assigned based on the execution time of the plan pair: if the execution time of the preceding target execution plan sample is shorter than that of the following target execution plan sample, the label is 1; conversely, if the execution time of the preceding target execution plan sample is longer than that of the following target execution plan sample, the label is 0. These labeled plan pairs constitute the sample set actually used for model training.

[0122] The BCE loss function is constructed based on six plan pairs, and its formula is as follows:

[0123]

[0124] Where N represents the number of planned pairs, y i The plan is to assign the true label (value 0 or 1) to i, and p i This is the probability that the model predicts plan i to be positive (label 1). This formula measures the difference between the model's prediction and the true label; the greater the difference, the greater the loss value.

[0125] During training, the model receives each plan pair as input, predicts the merits of the two objective execution plans in the plan pair, and outputs a probability value p between 0 and 1. i This represents the probability that the model believes the execution time of the previous execution plan sample is shorter (labeled 1). The loss value is then calculated using the BCE loss function, and the model parameters are adjusted using the backpropagation algorithm. If the prediction result differs significantly from the actual situation, for example, the model predicts execution plan A is faster (i.e., predicts p...), the model will adjust the parameters accordingly. i It is close to 1), but in actual execution, Plan B is executed faster (real label y). i If the value is 0, the model will undergo more extensive learning and correction.

[0126] See Figure 3D , Figure 3D This is a flowchart illustrating the model training method provided in an embodiment of this application. Figure 3D Show Figure 3A Step 104, which involves training an artificial intelligence model based on predicted performance data and actual performance data, can be achieved through steps 1041-1043.

[0127] In step 1041, the loss corresponding to each target execution plan sample is determined based on the actual performance data and predicted performance data of each target execution plan sample.

[0128] The loss for each target execution plan sample is determined based on the actual performance data and predicted performance data of each target execution plan sample. The explanation of the loss is similar to that above and will not be repeated here.

[0129] In step 1042, the weight corresponding to each target execution plan sample is determined, and based on the weight, the losses corresponding to multiple target execution plan samples are fused to obtain the total loss.

[0130] The total loss, obtained by merging the losses corresponding to multiple target execution plan samples based on weights, can be achieved as follows: The total loss is obtained by weighted summation of the losses corresponding to multiple target execution plan samples.

[0131] In some embodiments, determining the weight corresponding to each target execution plan sample can be achieved by performing the following processing for each target execution plan sample: obtaining the target fingerprint information of the target query statement sample corresponding to the target execution plan sample, and obtaining the fingerprint information corresponding to multiple candidate query statement samples respectively; determining the number of target fingerprint information included in the multiple fingerprint information, and determining the weight corresponding to the target execution plan sample based on the number, wherein the number and weight are non-negatively correlated.

[0132] The method for obtaining the target fingerprint information is similar to the method for obtaining the fingerprint information described above, and will not be repeated here.

[0133] Non-negative correlations include at least positive correlations and may also include constant correlations. A positive correlation indicates that the larger the quantity n, the larger the weight w; a constant correlation indicates that the weight does not change with the quantity. For example, non-negative correlations are shown in Table 1:

[0134] Table 1

[0135]

[0136] Thus, this embodiment of the application dynamically determines the weight of the execution plan samples based on the frequency of occurrence of target fingerprint information, thereby achieving intelligent weighting of training samples: execution plans corresponding to high-frequency query patterns receive higher weights to strengthen the model's learning of common scenarios, while execution plans corresponding to low-frequency queries retain appropriate weights to ensure that the model covers long-tail scenarios; this non-negative weighting mechanism based on fingerprint statistics avoids the problem of insufficient learning of important patterns caused by uniform sampling, and overcomes the risk of overfitting that may be caused by simple frequency weighting. Thus, while improving the model's ability to optimize mainstream queries, it maintains the generalization performance for dealing with diverse queries, and finally trains an artificial intelligence model that balances accuracy and robustness.

[0137] In some embodiments, the total loss is obtained by fusing the losses corresponding to multiple target execution plan samples based on weights. This can be achieved by: smoothing the weights corresponding to each target execution plan sample to obtain the target weights corresponding to each target execution plan sample; and fusing the losses corresponding to multiple target execution plan samples based on the target weights to obtain the total loss.

[0138] For example, the dynamic weight training mechanism based on query frequency introduces the sigmoid function to smooth the weight values ​​of high-frequency and low-frequency queries. The weights of low-frequency queries are not excessively weakened due to their low frequency, and they still play a role in model training, ensuring the effectiveness of low-frequency queries in model training. This helps the machine learning optimizer learn a wider range of query patterns, improves its generalization ability, and enables it to better adapt to various types of queries, thus improving overall performance.

[0139] In step 1043, the parameters of the artificial intelligence model are updated based on the total loss to obtain the trained artificial intelligence model.

[0140] Thus, the beneficial effects achieved by steps 1041-1043 in this embodiment are as follows: By using a dynamic weight training mechanism, different weights are set for different target execution plan samples. By dynamically allocating weights to each execution plan sample and fusing multi-sample losses, fine-grained control of the training process is achieved, enabling the artificial intelligence model to adaptively adjust the learning intensity for sample data of different importance and reliability. This weight mechanism can effectively reduce the interference of invalid samples and highlight the contribution of key samples. At the same time, loss fusion ensures that the model optimization direction takes into account both overall performance and individual differences, thereby significantly improving the prediction robustness and generalization ability of the artificial intelligence model in complex database environments, and ultimately generating a more accurate and reliable execution plan performance prediction model.

[0141] In some embodiments, after training the artificial intelligence model based on predicted performance data and actual performance data, new query statement samples and corresponding new execution plan samples are obtained; based on the new query statement samples and at least one target query statement sample, the new execution plan samples and candidate execution plan samples are evaluated to obtain evaluation information; when the evaluation information indicates that there is a difference between the new execution plan samples and candidate execution plan samples, the artificial intelligence model is trained again based on the new execution plan samples.

[0142] For example, the new query statement samples include new query statement sample 1 (corresponding to new execution plan sample 1) and new query statement sample 2 (corresponding to new execution plan sample 2). Based on new query statement sample 1 and at least one target query statement sample, the new execution plan sample 1 and the candidate execution plan sample are evaluated to obtain evaluation information 1. Based on new query statement sample 2 and at least one target query statement sample, the new execution plan sample 2 and the candidate execution plan sample are evaluated to obtain evaluation information 2. When evaluation information 1 indicates that there is a difference between new execution plan sample 1 and the candidate execution plan sample, and evaluation information 2 indicates that there is no difference between new execution plan sample 2 and the candidate execution plan sample, then new execution plan sample 2 is identified as redundant data, and the artificial intelligence model continues to be trained based on new execution plan sample 1.

[0143] Thus, by introducing an incremental learning mechanism, this application embodiment achieves continuous optimization and dynamic adaptation of the artificial intelligence model: it can automatically capture new query patterns (i.e., new query statement samples) and the differences between their execution plans and historical samples. When a significant difference is detected, the artificial intelligence model is retrained, enabling the model to continuously absorb new optimization strategies and performance characteristics. This effectively solves the problem of predictive performance degradation caused by changes in the database environment (such as data distribution drift) in traditional static models, thereby ensuring that the artificial intelligence model always maintains optimal decision accuracy. At the same time, it significantly reduces the cost of manual intervention and full retraining, filters out data with significant redundancy, significantly reduces the amount of data, and improves the efficiency of incremental training.

[0144] In some embodiments, based on a new query statement sample and at least one target query statement sample, a differential evaluation is performed on the new execution plan sample and the candidate execution plan sample to obtain evaluation information. This can be achieved by: obtaining new fingerprint information of the new query statement sample and obtaining fingerprint information corresponding to multiple candidate query statement samples respectively; determining the number of new fingerprint information included in the multiple fingerprint information; when the number is less than a second number threshold, the evaluation information is determined to indicate that there is a difference between the new execution plan sample and the candidate execution plan sample; when the new execution plan sample of the new query statement sample is inconsistent with the candidate execution plan sample of the target query statement sample, the evaluation information is determined to indicate that there is a difference between the new execution plan sample and the candidate execution plan sample.

[0145] The method for obtaining new fingerprint information is similar to that for obtaining fingerprint information mentioned above, and will not be repeated here.

[0146] For example, firstly, it is determined whether the new query sample is a high-frequency query (i.e., the number of new fingerprints included in multiple fingerprint information is greater than or equal to the second quantity threshold). If the new query sample is a high-frequency query, and the corresponding new execution plan sample is consistent with the candidate execution plan, it indicates that the artificial intelligence model has fully learned the characteristics of the new query sample, and also indicates that the underlying data distribution and query conditions have not changed significantly. At this time, the artificial intelligence model identifies the new query sample as redundant data (i.e., there is no difference between the new execution plan sample and the candidate execution plan sample), and directly skips the training stage, avoiding the additional computational resource consumption and time overhead caused by training on this type of data. For incremental data that does not meet the above conditions, i.e., low-frequency queries (i.e., the number of new fingerprints included in multiple fingerprint information is less than the second quantity threshold), changes in the plan enumeration results (i.e., the new execution plan sample of the new query sample is inconsistent with the candidate execution plan sample of the target query sample), etc., they will be filtered out and enter the subsequent training process.

[0147] As mentioned above, the model training method provided in this application can be implemented by various types of electronic devices, such as terminals, servers, or a combination of both. Therefore, the executing entity of each step will not be described again below. See Figure 6 , Figure 6 This is a flowchart illustrating the model training method provided in the embodiments of this application, combined with... Figure 6 The steps shown are explained.

[0148] In step 201, multiple execution plan samples are obtained, and the performance of each execution plan sample is predicted using an artificial intelligence model to obtain the predicted performance data of each execution plan sample.

[0149] Step 201 is similar to step 103 above, and will not be described again here.

[0150] It should be noted that the execution plan sample in step 201 can be a candidate execution plan sample of the above candidate query statement sample (i.e., an undeduplicated execution plan sample of the undeduplicated query statement sample), a target execution plan sample of the above candidate query statement sample (i.e., a deduplicated execution plan sample of the undeduplicated query statement sample), a candidate execution plan sample of the above target query statement sample (i.e., an undeduplicated execution plan sample of the deduplicated query statement sample), or a target execution plan sample of the above target query statement sample (i.e., a deduplicated execution plan sample of the deduplicated query statement sample).

[0151] In step 202, the actual performance data of each execution plan sample is obtained, and the loss corresponding to each execution plan sample is determined based on the actual performance data and the predicted performance data of each execution plan sample.

[0152] The method for determining the loss corresponding to each execution plan sample based on the actual performance data and predicted performance data of each execution plan sample is similar to the description of the loss above, and will not be repeated here.

[0153] In step 203, the weight corresponding to each execution plan sample is determined, and based on the weight, the losses corresponding to multiple execution plan samples are fused to obtain the total loss.

[0154] Step 203 is similar to step 1042 above, and will not be described again here.

[0155] In step 204, the parameters of the artificial intelligence model are updated based on the total loss to obtain the trained artificial intelligence model.

[0156] Step 204 is similar to step 1043 above, and will not be described again here.

[0157] In some embodiments, after training the artificial intelligence model based on predicted performance data and actual performance data, new query statement samples and corresponding new execution plan samples are obtained; based on the new query statement samples and the query statement samples, a differential evaluation is performed on the new execution plan samples and the execution plan samples to obtain evaluation information; when the evaluation information indicates that there is a difference between the new execution plan samples and the execution plan samples, the artificial intelligence model is further trained based on the new execution plan samples.

[0158] For example, the new query statement samples include new query statement sample 1 (corresponding to new execution plan sample 1) and new query statement sample 2 (corresponding to new execution plan sample 2). Based on new query statement sample 1 and the query statement sample, a differential evaluation is performed on new execution plan sample 1 and the execution plan sample to obtain evaluation information 1. Based on new query statement sample 2 and the query statement sample, a differential evaluation is performed on new execution plan sample 2 and the execution plan sample to obtain evaluation information 2. When evaluation information 1 indicates that there is a difference between new execution plan sample 1 and the execution plan sample, and evaluation information 2 indicates that there is no difference between new execution plan sample 2 and the execution plan sample, then new execution plan sample 2 is identified as redundant data, and the artificial intelligence model continues to be trained based on new execution plan sample 1.

[0159] Thus, by introducing an incremental learning mechanism, this application embodiment achieves continuous optimization and dynamic adaptation of the artificial intelligence model: it can automatically capture new query patterns (i.e., new query statement samples) and the differences between their execution plans and historical samples. When a significant difference is detected, the artificial intelligence model is retrained, enabling the model to continuously absorb new optimization strategies and performance characteristics. This effectively solves the problem of predictive performance degradation caused by changes in the database environment (such as data distribution drift) in traditional static models, thereby ensuring that the artificial intelligence model always maintains optimal decision accuracy. At the same time, it significantly reduces the cost of manual intervention and full retraining, filters out data with significant redundancy, significantly reduces the amount of data, and improves the efficiency of incremental training.

[0160] In some embodiments, a differential evaluation is performed on the new execution plan sample and the execution plan sample based on the new query statement sample and the query statement sample to obtain evaluation information. This can be achieved by: obtaining new fingerprint information of the new query statement sample and obtaining fingerprint information corresponding to multiple query statement samples respectively; determining the number of new fingerprint information included in the multiple fingerprint information; when the number is less than a second number threshold, the evaluation information is determined to indicate that there is a difference between the new execution plan sample and the execution plan sample; when the new execution plan sample of the new query statement sample is inconsistent with the execution plan sample of the query statement sample, the evaluation information is determined to indicate that there is a difference between the new execution plan sample and the execution plan sample.

[0161] The method for obtaining new fingerprint information is similar to that for obtaining fingerprint information mentioned above, and will not be repeated here.

[0162] For example, firstly, it is determined whether the new query sample is a high-frequency query (i.e., the number of new fingerprints included in multiple fingerprint information is greater than or equal to the second quantity threshold). If the new query sample is a high-frequency query, and the corresponding new execution plan sample is consistent with the execution plan sample, it indicates that the AI ​​model has fully learned the characteristics of the new query sample, and also indicates that the underlying data distribution and query conditions have not changed significantly. At this time, the AI ​​model identifies the new query sample as redundant data (i.e., there is no difference between the new execution plan sample and the execution plan sample), and directly skips the training stage, avoiding the additional computational resource consumption and time overhead caused by training on this type of data. For incremental data that does not meet the above conditions, i.e., low-frequency queries (i.e., the number of new fingerprints included in multiple fingerprint information is less than the second quantity threshold), changes in plan enumeration results (i.e., the new execution plan sample of the new query sample is inconsistent with the execution plan sample of the query sample), etc., they will be filtered out and enter the subsequent training process.

[0163] As mentioned above, the model training method provided in this application can be implemented by various types of electronic devices, such as terminals, servers, or a combination of both. Therefore, the executing entity of each step will not be described again below. See Figure 7 , Figure 7 This is a flowchart illustrating the model training method provided in the embodiments of this application, combined with... Figure 7 The steps shown are explained.

[0164] In step 301, historical query statement samples and corresponding historical execution plan samples are obtained. The performance of each historical execution plan sample is predicted by an artificial intelligence model to obtain the predicted performance data of each historical execution plan sample. Based on the predicted performance data and the actual performance data of each historical execution plan sample, the artificial intelligence model is trained.

[0165] The process of training the artificial intelligence model based on the predicted performance data and the actual performance data of each historical execution plan sample is similar to the process in step 104 above, and will not be repeated here.

[0166] It should be noted that the historical query statement sample in step 201 can be either the candidate query statement sample (i.e., the query statement sample that has not been deduplicated) or the target query statement sample (i.e., the query statement sample that has been deduplicated). The historical execution plan sample in step 202 can be either the candidate execution plan sample (i.e., the execution plan sample that has not been deduplicated) or the target execution plan sample (i.e., the execution plan sample that has been deduplicated).

[0167] In step 302, a sample of the new query statement and the corresponding sample of the new execution plan are obtained.

[0168] In step 303, based on the new query statement sample and the historical query statement sample, a differential evaluation is performed on the new execution plan sample and the historical execution plan sample to obtain evaluation information.

[0169] For example, the new query statement samples include new query statement sample 1 (corresponding to new execution plan sample 1) and new query statement sample 2 (corresponding to new execution plan sample 2). Based on new query statement sample 1 and historical query statement samples, a differential evaluation is performed on new execution plan sample 1 and historical execution plan samples to obtain evaluation information 1. Based on new query statement sample 2 and historical query statement samples, a differential evaluation is performed on new execution plan sample 2 and historical execution plan samples to obtain evaluation information 2. When evaluation information 1 indicates that there is a difference between new execution plan sample 1 and historical execution plan samples, and evaluation information 2 indicates that there is no difference between new execution plan sample 2 and historical execution plan samples, then new execution plan sample 2 is identified as redundant data, and the artificial intelligence model continues to be trained based on new execution plan sample 1.

[0170] Thus, by introducing an incremental learning mechanism, this application embodiment achieves continuous optimization and dynamic adaptation of the artificial intelligence model: it can automatically capture new query patterns (i.e., new query statement samples) and the differences between their execution plans and historical samples. When a significant difference is detected, the artificial intelligence model is retrained, enabling the model to continuously absorb new optimization strategies and performance characteristics. This effectively solves the problem of predictive performance degradation caused by changes in the database environment (such as data distribution drift) in traditional static models, thereby ensuring that the artificial intelligence model always maintains optimal decision accuracy. At the same time, it significantly reduces the cost of manual intervention and full retraining, filters out data with significant redundancy, significantly reduces the amount of data, and improves the efficiency of incremental training.

[0171] In some embodiments, a differential evaluation is performed on the new execution plan sample and the historical execution plan sample based on the new query statement sample and the historical query statement sample to obtain evaluation information. This can be achieved by: obtaining new fingerprint information of the new query statement sample and obtaining fingerprint information corresponding to multiple historical query statement samples respectively; determining the number of new fingerprint information included in the multiple fingerprint information; when the number is less than a second number threshold, the evaluation information is determined to indicate that there is a difference between the new execution plan sample and the historical execution plan sample; when the new execution plan sample of the new query statement sample is inconsistent with the historical execution plan sample of the historical query statement sample, the evaluation information is determined to indicate that there is a difference between the new execution plan sample and the historical execution plan sample.

[0172] The method for obtaining new fingerprint information is similar to that for obtaining fingerprint information mentioned above, and will not be repeated here.

[0173] For example, firstly, it is determined whether the new query sample is a high-frequency query (i.e., the number of new fingerprints included in multiple fingerprint information is greater than or equal to the second quantity threshold). If the new query sample is a high-frequency query, and the corresponding new execution plan sample is consistent with the candidate execution plan, it indicates that the artificial intelligence model has fully learned the characteristics of the new query sample, and also indicates that the underlying data distribution and query conditions have not changed significantly. At this time, the artificial intelligence model identifies the new query sample as redundant data (i.e., there is no difference between the new execution plan sample and the historical execution plan sample), and directly skips the training stage, avoiding the additional computational resource consumption and time overhead caused by training on this type of data. For incremental data that does not meet the above conditions, i.e., low-frequency queries (i.e., the number of new fingerprints included in multiple fingerprint information is less than the second quantity threshold), changes in plan enumeration results (i.e., the new execution plan sample of the new query sample is inconsistent with the historical execution plan sample of the historical query sample), etc., they will be filtered out and enter the subsequent training process.

[0174] In step 304, when the evaluation information characterizes a difference between the new execution plan sample and the historical execution plan sample, the artificial intelligence model is trained again based on the new execution plan sample.

[0175] The process of continuing to train the artificial intelligence model based on the new execution plan sample is similar to steps 103 and 104 above, and will not be repeated here.

[0176] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0177] In the field of database querying, to address the high overhead of actually executing the execution plan of Structured Query Language (SQL) during the training data construction process, cost-estimated training data is mainly used. The cost estimate (i.e., the performance data estimate) generated by the database optimizer replaces the actual execution result as the training data label, thus avoiding the time-consuming process of fully enumerating and actually executing the historical SQL query statements.

[0178] While these technologies can reduce resource consumption and time costs in actual execution to some extent, they have revealed several problems in practical applications. Firstly, the statistics from the database optimizer may be outdated due to untimely updates, failing to accurately reflect the current database status. Furthermore, in complex business scenarios, the insufficient modeling capabilities of the database optimizer can lead to significant discrepancies between cost estimates and actual execution costs, resulting in poor model training performance. Secondly, these technologies result in massive training data volumes, increasing the difficulty and cost of data processing and model training, and failing to meet the demands for efficient model training in real-world applications.

[0179] To address the aforementioned issues, this application provides a machine learning optimizer that combines a traditional database optimizer with a machine learning model (implemented by an AI model). The traditional database optimizer enumerates execution plans to generate a candidate plan set, while the machine learning model predicts and selects the optimal execution plan. This application also provides a model training method that enables rapid training of the machine learning optimizer. It performs two-layer data deduplication based on query fingerprints (i.e., the fingerprint information of the query statement) and the execution plan: fingerprint information is generated by parsing the SQL query statement to remove parameter differences while retaining core semantics, achieving semantic-level deduplication of SQL; the execution plan is modeled as a directed acyclic graph, and a graph hashing algorithm is used to achieve plan-level deduplication, reducing data redundancy and enumeration. Furthermore, this application introduces a dynamic weight allocation mechanism based on query frequency, assigning higher weights to high-frequency queries to focus on core scenarios, and providing weight compensation to low-frequency queries to balance diversity. For incremental training scenarios, training is focused on deduplication results and plan differences, reducing unnecessary training overhead from low-differentiation duplicate queries. By utilizing the above techniques, a rapid training method can be implemented in cold start and incremental training scenarios, which can synergistically improve training efficiency and model performance, provide more efficient and faster training capabilities, shorten the waiting time for users in the data collection and model training process, and reduce the system resource overhead generated by training.

[0180] The following is combined Figure 8 This application describes a two-layer data deduplication scheme based on query fingerprints and execution plans. This scheme effectively reduces training data redundancy and improves training efficiency and model performance by employing a two-layer deduplication strategy based on query fingerprints (i.e., the fingerprint information of the query statement) and execution plans. The two-layer data deduplication scheme specifically includes two parts: query fingerprint deduplication and plan deduplication, which will be described below.

[0181] 1) Query fingerprint deduplication

[0182] The fingerprint deduplication scheme consists of two parts: fingerprint calculation and sampling deduplication. The following sections will explain fingerprint calculation and sampling deduplication respectively.

[0183] 1.1) Fingerprint Calculation

[0184] For each historical SQL query (i.e., the candidate query sample mentioned above), lexical analysis is used to remove specific parameters from the historical query, retaining only the core semantic information. For example, for the historical queries “SELECT * FROM users WHERE age > 25” and “SELECT * FROM users WHERE age > 30”, after parameter removal using lexical analysis, the core semantic information is "select all records from the users table that satisfy age > a certain value". Subsequently, a hash algorithm, such as Message Digest Algorithm 5 (MD5) or Secure Hash Algorithm 256-bit (SHA-256), is used to calculate the parsed structure (i.e., semantic information) to generate a fingerprint (Digest) of the historical query as an identifier. This fingerprint information encompasses the core semantics of the historical query.

[0185] 1.2) Sampling and deduplication

[0186] Based on the generated fingerprint information, all historical query statements are grouped, with each group containing historical query statements sharing the same core semantics. If a group contains a large number of historical query statements (i.e., the number exceeds a threshold), random sampling is performed on these statements to ensure that each group retains a set number of historical query statements (e.g., a maximum of 3 to 10 different historical query statements). This results in deduplicated historical query statements (i.e., the target query statement sample mentioned above). This approach ensures sufficient samples are retained for subsequent training while effectively removing redundant data. Practical applications show that query fingerprint deduplication can eliminate over 95% of redundant query statements, significantly reducing unnecessary overhead caused by redundant queries.

[0187] 2) Deduplication Plan

[0188] The deduplication scheme consists of two parts: plan exploration and hash deduplication. The following sections will explain plan exploration and hash deduplication respectively.

[0189] 2.1) Exploration of the plan

[0190] After obtaining deduplicated historical query statements through query fingerprinting, to acquire diverse execution plans for model training, methods such as prompting intervention and on / off adjustments can be used to assist the database optimizer in generating a differentiated set of candidate plans (including candidate execution plans). This differentiated set of candidate plans refers to a series of distinct execution plans generated by the database optimizer for the same query statement, through prompting intervention and on / off adjustments. These execution plans differ in execution paths, resource usage, and operation order, and cover as many possible execution scenarios as possible, thereby increasing the probability of selecting the optimal execution plan.

[0191] Taking hint intervention as an example, hint intervention refers to explicitly adding database-specific hints to the deduplicated historical query statements. This allows for adjustments to the execution plan without altering the semantics of the deduplicated historical query statements. These hints provide explicit instructions to the database optimizer, directing it to select indexes, determine table join orders, and so on. For example, in MySQL, the index hint ` / *+INDEX(table_name index_name)* / ` can be used. When executing a complex query, if the database optimizer does not select an appropriate index, query performance will be poor. However, by adding this index hint, the database optimizer can be forced to use the index specified in the hint, thereby changing the execution plan and generating candidate execution plans. In MySQL, the hint ` / *+JOIN_PREFIX(t1,t2)* / ` instructs the database optimizer to perform join operations according to the order in which the tables appear, rather than having the database optimizer evaluate the join order itself. This method can precisely intervene in certain key aspects of the execution plan to obtain different execution paths, thus resulting in diversified execution plans.

[0192] Taking the switch adjustment method as an example, switch adjustment refers to adjusting the database optimizer switches. Optimizer switches are a series of pre-set parameters in the database system. By adjusting these parameters, the optimization strategy of the database optimizer can be changed, affecting its preference for operations such as join methods and scan methods. By reasonably adjusting these optimizer switch parameters, the database optimizer can be guided to generate different types of execution plans. Taking MySQL as an example, the system variable `optimizer_switch` controls the on / off state of various optimization strategies, used to dynamically enable or disable different optimization strategies of the query optimizer. Specifically, the `index_merge` variable controls whether the index merge optimization strategy is enabled. When `index_merge = on`, the index merge optimization strategy is enabled; when `index_merge = off`, the index merge optimization strategy is disabled. When index merging optimization is enabled, the database optimizer may merge multiple indexes to satisfy query conditions, resulting in a significantly different execution plan compared to when it is disabled. Similarly, the `block_nested_loop` and `batched_key_access` parameters are system variables in MySQL used to optimize table join operations, thus affecting query execution and performance optimization. The `block_nested_loop` parameter controls whether Block Nested Loop Join (BNLJ) is enabled, an algorithm that reduces the number of accesses to the inner table by caching rows from the outer loop. When this parameter is set to ON (the default value), MySQL reads rows from the outer table into memory blocks in batches when processing joins without suitable indexes, and then matches them with the inner table in batches, reducing disk I / O. The `batched_key_access` parameter optimizes index access efficiency. When enabled, MySQL uses a batch key access (BKA) strategy. BKA sends multiple query conditions to the storage engine in batches, leveraging the ordered nature of the index to prefetch data and reduce random I / O. Adjusting these two parameters can change MySQL's strategy for handling table joins, thus affecting the execution method of join operations and the overall execution plan.

[0193] 2.2) Hash Deduplication

[0194] The execution plans generated through the plan exploration operation (i.e., the aforementioned candidate execution plan samples) may contain a large number of duplicates, resulting in data redundancy. To address this issue, this embodiment models the generated execution plans as directed acyclic graphs (DAGs). In this modeling process, each operation step in the execution plan, such as scanning tables, filtering data, and joining tables, is abstracted as a node in the graph, and the dependencies between operations are represented as directed edges in the graph. Then, a unique hash value is calculated for each DAG using a graph hashing algorithm, which serves as the fingerprint information (PlanID) of the execution plan. Deduplication is performed based on the Plan ID, i.e., only execution plans with different Plan IDs are retained, resulting in deduplicated execution plans. This method can typically eliminate more than 70% of redundant plans.

[0195] By employing the aforementioned two-layer data deduplication process (i.e., query fingerprint deduplication and plan deduplication), redundant data can be rapidly eliminated from massive historical queries (including query statements and corresponding execution plans), significantly reducing the size of the actual execution plan set and lowering data collection time and costs. Simultaneously, the deduplication operation ensures the diversity between execution plans, enhancing the model's generalization ability and thus improving model training efficiency and effectiveness.

[0196] The following describes in detail the dynamic weight training mechanism based on query frequency provided in the embodiments of this application.

[0197] In training a machine learning optimizer, to enable the optimizer to better focus on key query statements while ensuring processing capabilities for various query types, this application introduces a dynamic weight training mechanism based on query frequency. After query fingerprint deduplication, the proportion of some high-frequency queries (including high-frequency query statements, i.e., queries that appear more frequently than a set threshold) in the dataset will be significantly reduced. However, these high-frequency queries often play a core role in actual business operations and still have a crucial impact on the model's prediction performance.

[0198] To enhance the machine learning optimizer's focus on high-frequency queries, a dynamic weight training mechanism based on query frequency assigns weights to different training data (including query sample and corresponding execution plan sample) by recording the frequency of different fingerprint information in historical data (i.e., historical query statements). Specifically, a comprehensive statistical analysis of historical queries is performed, calculating the occurrence count of each query fingerprint. Training data corresponding to query fingerprints with high occurrence frequency are assigned higher weights; conversely, training data corresponding to query fingerprints with low occurrence frequency are assigned lower weights. During model training, these weights determine the contribution of each training data point to the model's loss function. The loss function measures the difference between the model's predictions and the actual results. Training data with higher weights have a larger proportion in the loss function calculation, thus guiding the model to focus more on high-frequency queries during training and learn better execution plans for them. The embodiments of this application are not limited to the form of the loss function. Different loss functions can be selected for training according to actual needs and data characteristics. For example, it can be the mean squared error (MSE) loss function or the binary cross-entropy (BCE) loss function.

[0199] However, focusing solely on high-frequency queries can lead to poor performance of machine learning optimizers when faced with low-frequency queries, resulting in overfitting. This means the optimizer fits the high-frequency queries in the training data very well, but lacks generalization ability when dealing with low-frequency queries. To avoid this extreme scenario, a dynamic weight training mechanism based on query frequency introduces a sigmoid function to smooth the weight values ​​of both high-frequency and low-frequency queries. The weights of low-frequency queries are not excessively weakened due to their low frequency, and they still play a role in model training. This ensures the effectiveness of low-frequency queries in model training, helps the machine learning optimizer learn a wider range of query patterns, improves its generalization ability, and enables it to better adapt to various query types, thus improving overall performance.

[0200] The following is combined Figure 9 This application provides a detailed explanation of the differential training mechanism for incremental training scenarios.

[0201] For incremental training scenarios, in addition to utilizing deduplication and weight allocation mechanisms, this application embodiment improves incremental training efficiency by introducing differentiated training. By comparing the differences between incremental data and old data (i.e., existing data) in query fingerprints and execution plans, fine-grained selection of significantly differentiated training data is added to the incremental training set, filtering out data with significant redundancy to significantly reduce the data volume. This is specifically achieved through the following methods:

[0202] During the cold start phase, the model training system constructs and maintains two core data tables: the fingerprint grouping table (901) and the plan enumeration table (902). The fingerprint grouping table records the frequency of occurrence of different query fingerprints; the plan enumeration table stores the plan enumeration results (i.e., all candidate execution plans) for each query fingerprint or the plan enumeration results for each query statement—that is, all possible execution plans generated by the machine learning optimizer for each query statement. These two tables provide fundamental data support for subsequent incremental training data comparison and filtering.

[0203] After the model training system accumulates a batch of incremental data, it compares the differences between the old and new data based on the fingerprint grouping table and the plan enumeration table. First, it determines whether the query fingerprint in the incremental data belongs to a high-frequency query (e.g., frequency > 10 times). If a piece of incremental data belongs to a high-frequency query, and the corresponding plan enumeration result is consistent with the historical results recorded in the plan enumeration table (i.e., the plan enumeration result of the recorded query statement), it indicates that the machine learning optimizer has fully learned the characteristics of the query statement, and also indicates that the underlying data distribution and query conditions have not changed significantly. At this time, the model training system identifies the incremental data as redundant data and skips the training stage directly, avoiding the additional computational resource consumption and time overhead caused by training such data. For incremental data that does not meet the above conditions, i.e., low-frequency queries (e.g., frequency ≤ 10 times) and high-frequency queries with changed plan enumeration results, they will be filtered out and enter the subsequent training process.

[0204] In summary, at the data collection level, this application's embodiments employ accurate actual execution data to ensure that the training data truthfully and reliably reflects the actual operating conditions, effectively avoiding model training bias caused by estimation errors. In the data processing and training stages, this application's embodiments significantly reduce training data redundancy and substantially lower data collection and training costs through techniques such as two-layer data deduplication, dynamic weight allocation based on query frequency, and differentiated training in incremental training scenarios. Two-layer data deduplication effectively compresses the size of the plan set, dynamic weight allocation allows the model to focus on core scenarios while also considering diversity, and differentiated training in incremental training scenarios achieves efficient and accurate training. These techniques work synergistically to shorten training time, reduce system resource consumption, enhance model generalization ability, improve model training effects and performance, and provide an efficient and accurate solution for the rapid training of machine learning optimizers.

[0205] The exemplary application and implementation of the electronic device provided in the embodiments of this application have been used to describe the model training method provided in the embodiments of this application. The following will continue to describe the cooperation of the various modules in the model training device 555 provided in the embodiments of this application to implement the model training scheme.

[0206] The first deduplication module 5551 is used to obtain multiple candidate query statement samples for the database, and deduplicate the multiple candidate query statement samples to obtain at least one target query statement sample; the second deduplication module 5552 is used to generate multiple candidate execution plan samples for each target query statement sample, and deduplicate the multiple candidate execution plan samples to obtain at least one target execution plan sample; the prediction module 5553 is used to perform performance prediction on each target execution plan sample using an artificial intelligence model to obtain predicted performance data for each target execution plan sample; the training module 5554 is used to obtain actual performance data for each target execution plan sample, and train the artificial intelligence model based on the predicted performance data and the actual performance data.

[0207] In some embodiments, the first deduplication module 5551 is further configured to extract fingerprint information of each candidate query statement sample; based on the fingerprint information of the multiple candidate query statement samples, cluster the multiple candidate query statement samples to obtain multiple clusters, wherein each cluster includes at least one candidate query statement sample; and sample at least one target query statement sample from at least one cluster.

[0208] In some embodiments, the first deduplication module 5551 is further configured to perform semantic extraction on each candidate query statement sample to obtain semantic information of each candidate query statement sample; and perform hash processing on the semantic information of each candidate query statement sample to obtain fingerprint information of each candidate query statement sample.

[0209] In some embodiments, the first deduplication module 5551 is further configured to, when the number of candidate query statement samples included in any cluster is greater than or equal to a first quantity threshold, sample target query statement samples from any cluster with a number less than the first quantity threshold; and when the number of candidate query statement samples included in any cluster is less than the first quantity threshold, determine the candidate query statement samples included in any cluster as the target query statement samples.

[0210] In some embodiments, the second deduplication module 5552 is further configured to perform hash processing on each candidate execution plan sample to obtain fingerprint information of each candidate execution plan sample; and based on the fingerprint information of multiple candidate execution plan samples, to deduplicate the multiple candidate execution plan samples to obtain at least one target execution plan sample.

[0211] In some embodiments, the second deduplication module 5552 is further configured to construct a graph for each candidate execution plan sample to obtain a graph structure corresponding to each candidate execution plan sample; and to perform graph hashing on the graph structure corresponding to each candidate execution plan sample to obtain fingerprint information of each candidate execution plan sample.

[0212] In some embodiments, the second deduplication module 5552 is further configured to perform the following processing for each candidate execution plan sample: obtain at least one execution step of the candidate execution plan sample; determine each execution step as a node, and determine the edges between the nodes corresponding to the execution steps based on the dependencies between the execution steps; and construct a graph structure corresponding to the candidate execution plan sample based on the edges between the nodes and the nodes.

[0213] In some embodiments, the training module 5554 is further configured to determine the loss corresponding to each target execution plan sample based on the actual performance data and the predicted performance data of each target execution plan sample; determine the weight corresponding to each target execution plan sample, and based on the weight, fuse the losses corresponding to multiple target execution plan samples to obtain a total loss; and update the parameters of the artificial intelligence model based on the total loss to obtain the trained artificial intelligence model.

[0214] In some embodiments, the training module 5554 is further configured to perform the following processing for each target execution plan sample: obtain target fingerprint information of the target query statement sample corresponding to the target execution plan sample, and obtain fingerprint information corresponding to the plurality of candidate query statement samples respectively; determine the number of target fingerprint information included in the plurality of fingerprint information, and determine the weight corresponding to the target execution plan sample based on the number, wherein the number and the weight are non-negatively correlated.

[0215] In some embodiments, the training module 5554 is further configured to smooth the weights corresponding to each target execution plan sample to obtain target weights corresponding to each target execution plan sample; and based on the target weights, fuse the losses corresponding to multiple target execution plan samples to obtain a total loss.

[0216] In some embodiments, after training the artificial intelligence model based on the predicted performance data and the actual performance data, the training module 5554 is further configured to obtain new query statement samples and corresponding new execution plan samples; based on the new query statement samples and the at least one target query statement sample, perform a differential evaluation on the new execution plan sample and the candidate execution plan sample to obtain evaluation information; when the evaluation information indicates that there is a difference between the new execution plan sample and the candidate execution plan sample, continue training the artificial intelligence model based on the new execution plan sample.

[0217] In some embodiments, the training module 5554 is further configured to obtain new fingerprint information of the new query statement sample and obtain fingerprint information corresponding to the plurality of candidate query statement samples respectively; determine the number of new fingerprint information included in the plurality of fingerprint information; when the number is less than a second number threshold, determine the evaluation information as indicating that there is a difference between the new execution plan sample and the candidate execution plan sample; when the new execution plan sample of the new query statement sample is inconsistent with the candidate execution plan sample of the target query statement sample, determine the evaluation information as indicating that there is a difference between the new execution plan sample and the candidate execution plan sample.

[0218] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer program or computer-executable instructions from the computer-readable storage medium and executes the computer program or computer-executable instructions, causing the electronic device to perform the model training method described above in this application.

[0219] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the model training method provided in this application. For example, ... Figure 3A The model training method is shown.

[0220] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0221] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0222] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0223] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0224] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0225] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A model training method, characterized in that, The method includes: Obtain multiple candidate query statement samples for the database, and deduplicate the multiple candidate query statement samples to obtain at least one target query statement sample; For each target query statement sample, multiple candidate execution plan samples are generated, and the multiple candidate execution plan samples are deduplicated to obtain at least one target execution plan sample; The performance of each target execution plan sample is predicted using an artificial intelligence model, thereby obtaining the predicted performance data for each target execution plan sample. Obtain the actual performance data for each target execution plan sample, and train the artificial intelligence model based on the predicted performance data and the actual performance data.

2. The method according to claim 1, characterized in that, The process of deduplicating the multiple candidate query statement samples to obtain at least one target query statement sample includes: Extract the fingerprint information of each candidate query statement sample; Based on the fingerprint information of multiple candidate query statement samples, the multiple candidate query statement samples are clustered to obtain multiple clusters, wherein each cluster includes at least one candidate query statement sample. At least one target query statement sample is obtained from at least one of the said clusters.

3. The method according to claim 2, characterized in that, The step of extracting the fingerprint information of each candidate query statement sample includes: Semantic extraction is performed on each of the candidate query statement samples to obtain the semantic information of each candidate query statement sample; The semantic information of each candidate query statement sample is hashed to obtain the fingerprint information of each candidate query statement sample.

4. The method according to claim 2, characterized in that, The step of sampling at least one target query statement sample from at least one of the said clusters includes: When the number of candidate query statement samples included in any cluster is greater than or equal to a first quantity threshold, target query statement samples with a number less than the first quantity threshold are sampled from any cluster. When the number of candidate query statement samples included in any cluster is less than the first quantity threshold, the candidate query statement samples included in any cluster are determined as the target query statement samples.

5. The method according to claim 1, characterized in that, The step of deduplicating the multiple candidate execution plan samples to obtain at least one target execution plan sample includes: Each candidate execution plan sample is hashed to obtain the fingerprint information of each candidate execution plan sample; Based on the fingerprint information of multiple candidate execution plan samples, duplicates are removed from the multiple candidate execution plan samples to obtain at least one target execution plan sample.

6. The method according to claim 5, characterized in that, The step of hashing each candidate execution plan sample to obtain the fingerprint information of each candidate execution plan sample includes: For each candidate execution plan sample, a graph is constructed to obtain the graph structure corresponding to each candidate execution plan sample; Graph hashing is performed on the graph structure corresponding to each candidate execution plan sample to obtain the fingerprint information of each candidate execution plan sample.

7. The method according to claim 6, characterized in that, The step of constructing a graph for each candidate execution plan sample to obtain the graph structure corresponding to each candidate execution plan sample includes: For each of the candidate execution plan samples, the following processing is performed: Obtain at least one execution step of the candidate execution plan sample; Each execution step is identified as a node, and the edges between the nodes corresponding to the execution steps are determined based on the dependencies between the execution steps. Based on the edges between the nodes and the nodes themselves, a graph structure corresponding to the candidate execution plan sample is constructed.

8. The method according to claim 1, characterized in that, The step of training the artificial intelligence model based on the predicted performance data and the actual performance data includes: Based on the actual performance data and the predicted performance data of each target execution plan sample, determine the loss corresponding to each target execution plan sample; Determine the weight corresponding to each of the target execution plan samples, and based on the weights, fuse the losses corresponding to multiple target execution plan samples to obtain the total loss; The parameters of the artificial intelligence model are updated based on the total loss to obtain the trained artificial intelligence model.

9. The method according to claim 8, characterized in that, Determining the weight corresponding to each of the target execution plan samples includes: For each of the target execution plan samples, the following processing is performed: Obtain the target fingerprint information of the target query statement sample corresponding to the target execution plan sample, and obtain the fingerprint information corresponding to the multiple candidate query statement samples respectively; The number of target fingerprint information included in the plurality of fingerprint information is determined, and based on the number, the weight corresponding to the target execution plan sample is determined, wherein the number and the weight are non-negatively correlated.

10. The method according to claim 8, characterized in that, The total loss is obtained by fusing the losses corresponding to multiple target execution plan samples based on the weights, including: The weights corresponding to each target execution plan sample are smoothed to obtain the target weights corresponding to each target execution plan sample. Based on the target weights, the losses corresponding to multiple target execution plan samples are combined to obtain the total loss.

11. The method according to any one of claims 1-10, characterized in that, After training the artificial intelligence model based on the predicted performance data and the actual performance data, the method further includes: Obtain a sample of the new query statement and the corresponding sample of the new execution plan; Based on the new query statement sample and the at least one target query statement sample, a differential evaluation is performed on the new execution plan sample and the candidate execution plan sample to obtain evaluation information; When the evaluation information indicates a difference between the new execution plan sample and the candidate execution plan sample, the artificial intelligence model continues to be trained based on the new execution plan sample.

12. The method according to claim 11, characterized in that, The step involves evaluating the differences between the new execution plan sample and the candidate execution plan sample based on the new query statement sample and the at least one target query statement sample, to obtain evaluation information, including: Obtain new fingerprint information of the new query statement sample, and obtain fingerprint information corresponding to the multiple candidate query statement samples respectively; The number of new fingerprint information included in the plurality of fingerprint information is determined. When the number is less than a second number threshold, the evaluation information is determined to indicate that there is a difference between the new execution plan sample and the candidate execution plan sample. When the new execution plan sample of the new query statement is inconsistent with the candidate execution plan sample of the target query statement, the evaluation information is determined to indicate that there is a difference between the new execution plan sample and the candidate execution plan sample.

13. A model training device, characterized in that, The device includes: The first deduplication module is used to obtain multiple candidate query statement samples for the database, and to deduplicatize the multiple candidate query statement samples to obtain at least one target query statement sample. The second deduplication module is used to generate multiple candidate execution plan samples for each target query statement sample, and to deduplicatize the multiple candidate execution plan samples to obtain at least one target execution plan sample. The prediction module is used to predict the performance of each target execution plan sample using an artificial intelligence model, and obtain the predicted performance data of each target execution plan sample. The training module is used to acquire the actual performance data of each target execution plan sample, and to train the artificial intelligence model based on the predicted performance data and the actual performance data.

14. An electronic device, characterized in that, The electronic device includes: Memory is used to store computer programs or computer-executable instructions. A processor, when executing a computer program or computer-executable instructions stored in the memory, implements the model training method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The device stores a computer program or computer-executable instructions, which, when executed by a processor, implement the model training method according to any one of claims 1 to 12.

16. A computer program product comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by a processor, the model training method according to any one of claims 1 to 12 is implemented.

Images