Training method and device of artificial intelligence model, equipment, storage medium and computer program product

Abstract

The application provides a training method and device of an artificial intelligence model, equipment, a storage medium and a computer program product; the method comprises the following steps: acquiring a plurality of data processing tasks; grouping the plurality of data processing tasks to obtain a plurality of task groups, each task group comprising at least one data processing task; executing the plurality of task groups in parallel, in the process of executing each task group, for each data processing task in the task group, based on an execution plan included in the data processing task, executing a data processing statement included in the data processing task to obtain actual execution data of the data processing task; performing execution data prediction on each data processing task through an artificial intelligence model to obtain predicted execution data of each data processing task, and training the artificial intelligence model based on the actual execution data and the predicted execution data. Through the application, rapid training of the artificial intelligence model can be realized.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a training method, apparatus, device, storage medium, and computer program product for an artificial intelligence model. Background Technology

[0002] With the widespread application of database systems and the explosive growth of data volume, users have increasingly higher requirements for database performance. As a key component of the database management system, the optimizer is mainly responsible for analyzing and optimizing the data processing statements submitted by users. By selecting appropriate execution plans, such as determining the table join order, selecting appropriate indexes, and deciding whether to perform table scans, it improves the execution efficiency of data processing.

[0003] In related technologies, AI-based optimizers can combine data processing statements and AI models to generate specific execution plans. However, due to the high cost of acquiring training data, the promotion and deployment of AI-based optimizers in practical applications are somewhat limited. Summary of the Invention

[0004] This application provides a training method, apparatus, device, storage medium, and computer program product for an artificial intelligence model, which can achieve rapid training of the artificial intelligence model.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a method for training an artificial intelligence model, the method comprising:

[0007] Obtain multiple data processing tasks, wherein each data processing task includes a data processing statement for the database and at least one corresponding execution plan;

[0008] The multiple data processing tasks are grouped to obtain multiple task groups, wherein each task group includes at least one data processing task.

[0009] The multiple task groups are executed in parallel. During the execution of each task group, for each data processing task in the task group, based on the execution plan included in the data processing task, the data processing statements included in the data processing task are executed to obtain the actual execution data of the data processing task.

[0010] The artificial intelligence model is used to predict the execution data of each data processing task to obtain the predicted execution data of each data processing task. Based on the actual execution data and the predicted execution data, the artificial intelligence model is trained.

[0011] This application provides a training device for an artificial intelligence model, the device comprising:

[0012] The task extraction module is used to obtain multiple data processing tasks, wherein each data processing task includes a data processing statement for the database and at least one corresponding execution plan;

[0013] The task grouping module is used to group the multiple data processing tasks to obtain multiple task groups, wherein each task group includes at least one data processing task.

[0014] The task execution module is used to execute the multiple task groups in parallel. During the execution of each task group, for each data processing task in the task group, based on the execution plan included in the data processing task, the data processing statements included in the data processing task are executed to obtain the actual execution data of the data processing task.

[0015] The model training module is used to predict the execution data of each data processing task through the artificial intelligence model, obtain the predicted execution data of each data processing task, and train the artificial intelligence model based on the actual execution data and the predicted execution data.

[0016] This application provides an electronic device, the electronic device comprising:

[0017] Memory is used to store executable instructions or computer programs.

[0018] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the training method for the artificial intelligence model provided in the embodiments of this application.

[0019] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the training method of the artificial intelligence model provided in this application when executed by a processor.

[0020] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the training method for the artificial intelligence model provided in this application.

[0021] The embodiments of this application have the following beneficial effects:

[0022] By grouping multiple data processing tasks and executing data processing statements in parallel based on the execution plan of each group's data processing tasks, the overall execution efficiency of multiple data processing tasks is improved. On this basis, the execution data of each data processing task is predicted by an artificial intelligence model, and the actual execution data is compared with the predicted execution data to train the artificial intelligence model. This allows for full utilization of a large amount of actual execution data as feedback, rapid adjustment and optimization of model parameters, thereby achieving rapid training of the artificial intelligence model. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the architecture of the training system for the artificial intelligence model provided in this application embodiment;

[0024] Figure 2 This is a schematic diagram of the structure of an electronic device for training an artificial intelligence model provided in an embodiment of this application;

[0025] Figure 3 This is a first flowchart illustrating the training method for the artificial intelligence model provided in this application embodiment;

[0026] Figure 4 This is a second flowchart illustrating the training method for the artificial intelligence model provided in this application embodiment;

[0027] Figure 5 This is a schematic diagram of the third process of the training method for the artificial intelligence model provided in the embodiments of this application;

[0028] Figure 6 This is a schematic diagram of the fourth process of the training method for the artificial intelligence model provided in the embodiments of this application;

[0029] Figure 7 This is a schematic diagram of the fifth process of the training method for the artificial intelligence model provided in the embodiments of this application;

[0030] Figure 8 This is a schematic diagram illustrating the first principle of the training method for the artificial intelligence model provided in this application embodiment;

[0031] Figure 9 This is a schematic diagram illustrating the second principle of the training method for the artificial intelligence model provided in this application embodiment;

[0032] Figure 10 This is a schematic diagram of the third principle of the training method for the artificial intelligence model provided in the embodiments of this application;

[0033] Figure 11 This is a schematic diagram of the fourth principle of the training method for the artificial intelligence model provided in the embodiments of this application;

[0034] Figure 12This is a schematic diagram of the fifth principle of the training method for the artificial intelligence model provided in the embodiments of this application.

[0035] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0038] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0039] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0040] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0041] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0042] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0043] 1) Responding to: used to indicate the conditions or states on which the operation is performed depends. When the conditions or states on which it depends are met, one or more operations can be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0044] 2) Data Processing Statements: Data processing statements, often simply called statements, are a form of business logic expression that adheres to a certain declarative programming language specification. They are typically used to describe operations and processing of data, such as querying, updating, inserting, or deleting data. Before executing a data processing statement, it needs to be parsed and validated according to the data definition. The data definition is the necessary semantic association information during statement execution; it describes the structure, type, constraints, and other attributes of the data. This information is crucial to ensuring the correctness and validity of the data processing statement.

[0045] 3) Execution Plan: An execution plan refers to the specific operational steps and strategies that a database system devises to execute a data processing statement. The execution plan instructs the database to access table data in specific ways, including using specific indexes, employing specific table join methods (such as nested loop joins or hash joins), and specific data filtering and sorting steps. The execution plan directly impacts query execution efficiency. The same data processing statement may have multiple candidate execution plans, and there may be significant performance differences between different candidate execution plans. The optimizer combines the data processing statement and statistical data to generate several candidate execution plans and selects the one with the lowest estimated cost to hand over to the executor for execution.

[0046] 4) Basic Data: Basic data refers to the original data involved in the execution of data processing statements. Basic data is the object of statement processing and operation and adheres to certain semantic constraints. Basic data can be real business data actually generated and used in business operations, such as user information and transaction records, or it can be simulated business data constructed based on the characteristics of real business data.

[0047] 5) Statistical data: Statistical data is descriptive information about basic data. For example, statistical data can include key features such as the number of rows in a data table, the data distribution model, and the correlation between columns. These statistical data are usually obtained by sampling and analyzing real business data and are used to evaluate the efficiency and cost of the execution plan. At the same time, the evaluation can also be directly supported by data samples, which is called sampling statistics.

[0048] 6) Estimated data: Estimated data is the quantified overhead data obtained by the optimizer in evaluating the execution plan. At this time, the data processing statements may not have been actually executed, or they may have undergone lightweight pre-execution.

[0049] 7) Execution data: Execution data refers to the specific information generated during the actual operation of an execution plan, reflecting the performance of the execution plan in actual operation. For example, execution data may include the completion time of the execution plan, the usage time of the central processing unit (CPU), and the waiting time.

[0050] 8) Pre-training: Pre-training refers to a series of processing and analysis of data before formally training an artificial intelligence model to generate training data for training the artificial intelligence model. For example, analyzing data processing statements to obtain several candidate execution plans, and then executing the data processing statements on the basic data (or samples) for each candidate execution plan to obtain the corresponding estimated data and execution data. Each unit of data (i.e. training data) in the complete data generated by pre-training includes the data processing statement, the execution plan adopted, the estimated data, and the number of executions.

[0051] 9) Training: Training refers to the process by which algorithm engineers encode and process the complete data generated in the pre-training phase, write algorithms for model training, and then train the model using computing resources (usually dedicated GPU clusters or cloud services) to ultimately generate an artificial intelligence (AI) model. In this process, engineers will convert the data into a format that the model can process, design appropriate training algorithms, and use powerful computing resources to optimize the model's parameters to improve the model's performance and accuracy.

[0052] 10) Optimizer: The optimizer is a key component of a database management system (DBMS). Its main function is to generate multiple candidate execution plans based on data processing statements and statistical data, and select the execution plan with the lowest estimated cost to return to the executor. Some optimizers also have pre-execution capabilities, that is, to perform preliminary execution on the candidate plans before actual execution and to dynamically prune them based on the pre-execution results to further optimize the execution plan.

[0053] 11) AI Optimizer: An AI optimizer is an optimizer that integrates artificial intelligence models. It has the ability to call AI models and adopt the output results of the models. The AI ​​optimizer can integrate multiple types of models. Each model has its own applicable conditions, input parameters and evaluation results. The AI ​​optimizer calls the appropriate model to complete the work when needed. For example, some models are only used to replace statistical data and generate estimation results, while others can directly make optimization decisions, and still others can generate self-contained execution plans.

[0054] 12) Database Management Service: The database management service is responsible for scheduling and delivering database instances. It also provides functions such as real-time monitoring, data backup, and database cloning.

[0055] 13) Production load: Production load is the actual business traffic, which consists of several data processing statement sequences. The data processing statement sequences are executed concurrently. Each data processing statement sequence contains several data processing statements executed sequentially. The data processing statements in the production load can appear repeatedly in different sequences or in the same sequence. Each time they appear, the optimal solution is evaluated and executed once. The production load is carried by database instances in the production cluster. The basic data in these database instances is called real business data or production data.

[0056] 14) Pre-training Load: A pre-training load refers to a set of data processing tasks used to generate training data. The data processing statements in a pre-training load are accompanied by explicitly specified execution plans. Unlike production loads, a single data processing statement in a pre-training load may have multiple candidate execution plans, and each plan will be executed once. The pre-training load is carried by database instances (pre-training instances) in the pre-training cluster. The data in the database instance can be complete production data or data sampled from production data. With effective resource isolation measures, database instances can even share the same data as production instances to improve resource utilization efficiency.

[0057] 15) Training Job: A training job is a comprehensive task unit. A training job includes pre-training load, specification description of pre-training instances, time budget and resource budget, and other constraints. The pre-training cluster and the training cluster complete the work under the coordination of the job management service.

[0058] 16) AI Model Repository: An AI model repository is a storage device or platform specifically designed for storing and managing trained artificial intelligence models.

[0059] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for training an artificial intelligence model, which can enable rapid training of an artificial intelligence model for data processing.

[0060] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the training system for the artificial intelligence model provided in this application embodiment. In the training system 10 for the artificial intelligence model provided in this application embodiment, in order to support the training application of an artificial intelligence model, the terminal 400 connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0061] Terminal 400 can be used to obtain training requests for artificial intelligence models that carry multiple data processing tasks.

[0062] In some embodiments, a training plugin for an artificial intelligence model can be embedded in the client running in the terminal 400 to implement the training method of the artificial intelligence model locally on the client. For example, the terminal 400 calls the training plugin for the artificial intelligence model to implement the training method of the artificial intelligence model, obtains multiple data processing tasks, groups the multiple data processing tasks to obtain multiple task groups, and executes multiple task groups in parallel. During the execution of each task group, for each data processing task in the task group, based on the execution plan included in the data processing task, the data processing statements included in the data processing task are executed to obtain the actual execution data of the data processing task. The execution data of each data processing task is predicted by the artificial intelligence model to obtain the predicted execution data of each data processing task. Based on the actual execution data and the predicted execution data, the artificial intelligence model is trained.

[0063] It is worth noting that the trained artificial intelligence model can be stored locally on the terminal 400 and retrieved when needed.

[0064] In some embodiments, after the terminal 400 obtains a training request for an artificial intelligence model carrying multiple data processing tasks, it calls the training interface of the artificial intelligence model of the server 200 (which can be provided as a cloud service, i.e., an artificial intelligence model training service). The server 200 implements the training method of the artificial intelligence model through the training plugin of the artificial intelligence model, groups the multiple data processing tasks into multiple task groups, and executes the multiple task groups in parallel. During the execution of each task group, for each data processing task in the task group, based on the execution plan included in the data processing task, the data processing statements included in the data processing task are executed to obtain the actual execution data of the data processing task. The execution data of each data processing task is predicted by the artificial intelligence model to obtain the predicted execution data of each data processing task. Based on the actual execution data and the predicted execution data, the artificial intelligence model is trained.

[0065] It is worth noting that the trained artificial intelligence model can be stored locally on server 200 and retrieved when needed, or it can be returned to terminal 400.

[0066] In some embodiments, server 200 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0067] Terminal 400 can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0068] See Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device for training an artificial intelligence model provided in an embodiment of this application. Figure 2 The electronic device 500 shown can be Figure 1 The terminal 400 or server 200, and the electronic device 500 include: at least one processor 510, memory 550, and at least one network interface 520. The various components in server 200 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 2 The general labeled all buses as Bus System 540.

[0069] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0070] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls;

[0071] In some embodiments, when the terminal 400 independently completes the training task of the artificial intelligence model, the server 200 provided in this application embodiment does not include the user interface 530.

[0072] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 510.

[0073] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0074] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0075] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0076] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0077] Presentation module 553 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with user interface 530;

[0078] In some embodiments, when the terminal 400 independently completes the training task of the artificial intelligence model, the server 200 provided in this application embodiment may not include the presentation module 553.

[0079] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532; in some embodiments, when the terminal 400 independently completes the training task of the artificial intelligence model, the server 200 provided in this application embodiment may not include the presentation module 553.

[0080] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A training device 555 for an artificial intelligence model stored in memory 550 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a task extraction module 5551, a task grouping module 5552, a task execution module 5553, and a model training module 5554. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0081] In other embodiments, the apparatus provided in this application can be implemented in hardware. For example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the training method of the artificial intelligence model provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0082] As mentioned above, the electronic device implementing the training method of the artificial intelligence model in the embodiments of this application can be a terminal, a server, or a combination of both. Therefore, the executing entity of each step will not be described again below. See [link to relevant documentation]. Figure 3 , Figure 3 This is a first flowchart illustrating the training method for the artificial intelligence model provided in this application embodiment, which will be combined with... Figure 3 The steps shown are explained.

[0083] In step 101, multiple data processing tasks are acquired.

[0084] Each data processing task includes data processing statements for the database and at least one corresponding execution plan.

[0085] As an example, a data processing task refers to any one of the following operations on a database: Create, Delete, Update, or Select. Create refers to adding new data to the database, Delete refers to deleting existing data from the database, Update refers to modifying existing data in the database, and Select refers to retrieving data from the database. The database can be a structured database (SQL database) or a key-value database (KV database), without any limitation here.

[0086] Each data processing task includes a data processing statement for the database and at least one corresponding execution plan. The data processing statement refers to the instruction submitted to the database, and the execution plan refers to the specific operation steps and strategies generated by the database system to execute the data processing statement. For example, for an SQL database, the data processing statement may be "SELECT * FROM users WHERE age>18", and the execution plan may be "first scan the users table, and then filter out data with age greater than 18".

[0087] Multiple data processing tasks can be obtained manually, through automated tools or scripts, or by sampling data processing tasks in the production environment. There are no restrictions here. The following example will be used to illustrate how to obtain multiple data processing tasks by sampling data processing tasks in the production environment of the database.

[0088] In some embodiments, see Figure 4 , Figure 4 This is a schematic diagram of the second process of the training method for the artificial intelligence model provided in the embodiments of this application. Figure 3 Step 101 shown can be implemented through steps 1011 to 1012, which are explained in detail below.

[0089] In step 1011, in the production environment of the database, at least one historical data processing statement corresponding to the production instance is obtained.

[0090] The production environment refers to the actual operating database system environment, and the production instance refers to the database instance running in that environment.

[0091] As an example, in response to an abnormal load change in any production instance in the database production environment, at least one historical data processing statement corresponding to the production instance with the abnormal load change is obtained. Load refers to the access pressure and resource consumption that the database instance bears during operation. An abnormal load change means that the load borne by the database instance during operation experiences a sharp and abnormal fluctuation or change. The abnormal load change of the production instance can be judged by the preset load change rules. For example, an abnormal load change could be that the CPU utilization of any production instance suddenly rises from less than 20% to more than 80% within 5 minutes.

[0092] Alternatively, a timed mechanism can be used to periodically retrieve at least one historical data processing statement corresponding to the production instance; this is not limited here.

[0093] In some embodiments, the "obtaining at least one historical data processing statement corresponding to the production instance" in step 1011 can be achieved through the following process: when the historical execution record of the production instance is extracted from the production log of the database, at least one historical data processing statement is selected from the historical execution record; when the historical execution record of the production instance is not extracted from the production log of the database, the execution record of the production instance is collected in real time, and at least one historical data processing statement is selected from the collected execution record.

[0094] The historical execution record can include multiple data processing statements, as well as the historical execution plan and historical execution data for each data processing statement. The execution data refers to the specific overhead information generated during the execution of data processing statements based on the execution plan, including: the completion time of the execution plan, the CPU usage time, and the CPU waiting time.

[0095] As an example, database production logs refer to various records generated during database operation. These logs can be database audit logs or database performance history records (e.g., Oracle AWR), without limitation. The process involves extracting historical execution records of the production instance from the production logs. Historical execution records are collections of data processing statements executed by the database instance during operation. Based on a filtering strategy, at least one historical data processing statement is selected from these records. This filtering strategy can be based on a time interval, selecting at least one historical data processing statement executed within a preset time interval; or it can be based on the number of executions, selecting at least one historical data processing statement whose execution count exceeds a threshold. This threshold can be preset by engineers based on their experience or predicted by a corresponding neural network model, without limitation.

[0096] When no historical execution records of the production instance are extracted from the production logs of the database, the newly added execution records of the production instance within a preset time period starting from the time when "no historical execution records of the production instance are extracted" are collected, and at least one historical data processing statement is selected from the collected execution records. The processing of "selecting at least one historical data processing statement from the collected execution records" can be referred to the processing of "selecting at least one historical data processing statement from historical execution records" above, and will not be repeated here.

[0097] In this embodiment, by extracting historical execution records from the database production logs or collecting execution records in real time, not only is the existing production log information fully utilized, but real-time data can also be supplemented in a timely manner when the production logs are insufficient, thereby ensuring the integrity and accuracy of the data processing tasks. In addition, through flexible filtering strategies, representative historical data processing statements can be filtered out according to different needs (e.g., time interval, number of executions, etc.), further improving the targeting and effectiveness of historical data processing statement filtering.

[0098] In step 1012, for each historical data processing statement, at least one corresponding candidate execution plan is generated, and the historical data processing statement and the corresponding at least one candidate execution plan are combined to obtain the data processing task.

[0099] As an example, for each historical data processing statement, at least one candidate execution plan is generated using a database optimizer or execution plan generation tool. The historical data processing statement and the corresponding candidate execution plan are then combined to obtain a data processing task, which includes the data processing statement and the corresponding execution plan.

[0100] For example, for the historical data processing statement "Data Processing Statement A", three corresponding candidate execution plans "Plan A", "Plan B" and "Plan C" are generated. The data processing task obtained by combining the historical data processing statement "Data Processing Statement A" and the three candidate execution plans "Plan A", "Plan B" and "Plan C" can be represented as {Data Processing Statement: "Data Processing Statement A", Execution Plan: ["Plan A", "Plan B", "Plan C"]}}.

[0101] In the process of generating at least one corresponding candidate execution plan for each historical data processing statement, the historical execution plan of the historical data processing statement can be obtained from the historical execution record, and the historical execution plan can be directly determined as one of the candidate execution plans of the historical data processing statement.

[0102] In this embodiment, based on each historical data processing statement in the production instance of the database production environment, at least one corresponding candidate execution plan is quickly generated. The historical data processing statements and candidate execution plans are then combined to form a data processing task. This fully utilizes the actual running data in the production environment, ensuring that the generated data processing task is highly consistent with the actual operating conditions. At the same time, when the production instance experiences abnormal load changes, relevant historical data processing statements are obtained, thereby quickly locating the key statements that may cause performance problems. This provides a clear optimization direction for the subsequent training of the artificial intelligence model. In addition, new candidate execution plans are generated through the database optimizer or execution plan generation tool, enriching the selection range of execution plans and providing a rich data foundation for the subsequent training of the artificial intelligence model.

[0103] In step 102, multiple data processing tasks are grouped to obtain multiple task groups.

[0104] As an example, a target grouping strategy is obtained, and multiple data processing tasks are grouped based on the target grouping strategy to obtain multiple task groups, each task group including at least one data processing task.

[0105] In some embodiments, see Figure 5 , Figure 5 This is a schematic diagram of the third process of the training method for the artificial intelligence model provided in the embodiments of this application. Figure 3Step 102 shown can be implemented through steps 1021 to 1022, which are explained in detail below.

[0106] In step 1021, a target grouping strategy for grouping multiple data processing tasks is determined from multiple grouping strategies.

[0107] The grouping strategies include strategies that group by clustering and strategies that group by decoupling.

[0108] As an example, the workload of the worker node cluster used to execute multiple task groups is obtained. The workload includes CPU utilization, memory usage, disk I / O operation frequency, etc. Based on the workload of the worker node cluster, a target grouping strategy for grouping multiple data processing tasks is determined from multiple grouping strategies. When the workload of the worker node cluster is higher than the workload threshold, the strategy of grouping by clustering is determined as the target grouping strategy; when the workload of the worker node cluster is lower than the workload threshold, the strategy of grouping by decoupling is determined as the target grouping strategy. The workload threshold can be preset by engineers based on engineering experience, or it can be predicted based on a corresponding neural network model, and is not limited here.

[0109] In step 1022, multiple data processing tasks are grouped based on the target grouping strategy to obtain multiple task groups.

[0110] As an example, when the target grouping strategy is to group by clustering, multiple data processing tasks are clustered to obtain multiple task groups; when the target grouping strategy is to group by decoupling, multiple data processing tasks are decoupled to obtain multiple task groups.

[0111] In this embodiment of the application, by dynamically selecting a grouping strategy, data processing tasks can be flexibly grouped according to the actual workload of the worker node cluster. When the workload of the worker node cluster is high, clustering can be used to group tasks with similar resource requirements, thereby improving resource utilization and execution efficiency. When the workload is low, decoupling can be used to group tasks with different resource requirements, thereby avoiding resource competition and ensuring the priority execution of critical tasks.

[0112] In some embodiments, see Figure 6 , Figure 6 This is a schematic diagram of the fourth process of the training method for the artificial intelligence model provided in the embodiments of this application. Figure 5 Step 1022 shown can be implemented through steps 10221A to 10222A, which are explained in detail below.

[0113] In step 10221A, when the target grouping strategy is a strategy of grouping by clustering, the resources required to execute each data processing task are obtained.

[0114] As an example, when the target grouping strategy is a strategy of grouping by clustering, the resources required to execute each data processing task are obtained. The resources can be data resources, such as the data involved in the execution of the data processing task, or computing resources, such as the CPU resources and memory resources required to execute each data processing task.

[0115] In step 10222A, based on the resources required by each data processing task, multiple data processing tasks are clustered to obtain multiple clusters, and the multiple clusters are determined as multiple task groups.

[0116] As an example, based on the resources required for each data processing task, multiple data processing tasks are clustered, resulting in multiple clusters that are then identified as task groups. Each task group includes data processing tasks with the same resource requirements, or the difference in resource requirements between any two data processing tasks within each task group is less than a first difference threshold. This first difference threshold can be preset by engineers based on their engineering experience, or it can be predicted based on a corresponding neural network model; no specific limitation is imposed here. For example, when the resources are data resources, data processing tasks involving the same data table or the same index can be grouped into the same group.

[0117] Depending on the type of resources, the method for determining the differences in resources required by two data processing tasks also differs. When the resources are data resources, the differences can be determined by comparing the overlap between the data tables and indexes involved in the data processing tasks. When the resources are hardware resources, the differences can be determined by comparing the CPU, memory, I / O operation frequency, and network bandwidth requirements of the two data processing tasks.

[0118] In this embodiment, a clustering-based grouping strategy can be used to accurately group tasks according to the resources required by each data processing task. When the resources are data resources, tasks involving the same data table or index are grouped together, which can reduce data access overhead and reduce I / O operation frequency, thereby improving the overall performance of the system. At the same time, by ensuring that the difference in resource requirements of tasks within each task group is lower than a first difference threshold, context switching and resource contention can be further reduced, improving the efficiency and stability of task execution.

[0119] In some embodiments, step 10222A can be implemented by the following process: selecting multiple target resources from the resources required by multiple data processing tasks; using the multiple target resources as cluster centers to cluster the multiple data processing tasks to obtain multiple clusters.

[0120] Among them, the difference between target resources is greater than the difference threshold. The difference threshold can be preset by engineering technicians based on their engineering experience, or it can be predicted based on the corresponding neural network model. There is no limitation here.

[0121] As an example, from multiple resources required by multiple data processing tasks, a first target resource is randomly selected. From the resources other than the target resource, the resource with the largest difference from the first target resource is selected and designated as the second target resource. From the resources other than the target resource, the resource with the largest average difference from the target resource is selected and designated as the third target resource, and so on. Under the premise of ensuring that the difference between the target resources is greater than the difference threshold, multiple target resources are selected from the resources required by multiple data processing tasks. These multiple target resources are used as cluster centers to cluster the multiple data processing tasks, resulting in multiple clusters.

[0122] In this embodiment, by progressively screening target resources, it is ensured that the difference between each newly added cluster center and the existing cluster centers is greater than the difference threshold. The target resources are then used as cluster centers to cluster data processing tasks, ensuring the rationality and effectiveness of the clustering. This not only efficiently groups data processing tasks with similar resource requirements but also ensures that the resource requirements of data processing tasks in different clusters are sufficiently different, thereby avoiding resource competition and improving resource utilization and the execution efficiency of data processing tasks.

[0123] In some embodiments, see Figure 7 , Figure 7 This is a schematic diagram of the fifth step of the training method for the artificial intelligence model provided in this application embodiment. Figure 5 Step 1022 shown can be implemented through steps 10221B to 10222B, which are explained in detail below.

[0124] In step 10221B, when the target grouping strategy is a strategy of grouping by decoupling, the resources required to execute each data processing task are obtained.

[0125] As an example, when the target grouping strategy is a strategy of grouping in a decoupling manner, the resources required to execute each data processing task are obtained. The resources can be data resources, such as the data involved in the execution of the data processing task, or computing resources, such as the CPU resources and memory resources required to execute each data processing task.

[0126] In step 10222B, based on the resources required by each data processing task, multiple data processing tasks are grouped according to resource coupling relationships to obtain multiple task groups.

[0127] As an example, the resources required for each data processing task within a task group are different, or the difference in resources required between any two data processing tasks within a task group exceeds a second difference threshold. This second difference threshold can be preset by engineers based on their engineering experience, or it can be predicted based on a corresponding neural network model; no specific limitation is imposed here. For instance, when the resources are data resources, data processing tasks involving the same data table or the same index can be divided into different task groups.

[0128] In the embodiments of this application, by using a grouping strategy based on resource coupling relationships, data processing tasks with large differences in resource requirements can be effectively decoupled and grouped. This ensures that the resource requirements of tasks within each task group are different or highly different, thereby avoiding resource competition and improving the overall performance and stability of the system. For example, when the resource is a data resource, dividing data processing tasks involving the same data table or the same index into different task groups can reduce data access conflicts and optimize I / O operations.

[0129] In some embodiments, step 10222B can be implemented by the following process: based on the resources required for each data processing task, determine the data processing tasks with resource coupling relationships and the data processing tasks without resource coupling relationships from multiple data processing tasks; decouple the data processing tasks with resource coupling relationships to obtain decoupled data processing tasks, and divide the decoupled data processing tasks and the data processing tasks without resource coupling relationships into corresponding task groups.

[0130] Within each task group, there is no resource coupling between the data processing tasks.

[0131] As an example, based on the resources required for each data processing task, data processing tasks whose required resources are different from those of other data processing tasks are identified as data processing tasks without resource coupling. Data processing tasks with the same required resources or whose required resources differ by less than a third difference threshold are identified as data processing tasks with resource coupling. While ensuring that data processing tasks with resource coupling are assigned to different task groups, data processing tasks with resource coupling and those without resource coupling are grouped. The third difference threshold can be preset by engineers based on their engineering experience or predicted based on a corresponding neural network model, and is not limited here.

[0132] For example, when the resource is a data resource, the task group includes data processing tasks {Task 1, Task 2, Task 3, Task 4, Task 5}. Executing Task 1, Task 2, and Task 3 all require calling data table A, executing Task 4 requires calling data table B, and executing Task 5 requires calling data table C. Tasks 4 and 5, whose required resources are different from those of the other data processing tasks, are identified as data processing tasks without resource coupling. Tasks 1, Task 2, and Task 3, which require the same resources, are identified as data processing tasks with resource coupling. Under the premise of ensuring that data processing tasks with resource coupling are divided into different task groups, {Task 1, Task 2, Task 3, Task 4, Task 5} is grouped into three groups: {Task 1, Task 4}, {Task 2, Task 5}, and {Task 3}.

[0133] In this embodiment, through detailed resource coupling analysis, efficient decoupling and reasonable grouping of data processing tasks are achieved, ensuring that there is no resource coupling between tasks within each task group, thereby effectively avoiding resource competition and improving the system's concurrent processing capability and response speed. At the same time, for data processing tasks with unique resource requirements and no resource coupling, they are directly assigned to the corresponding task groups, simplifying the grouping logic.

[0134] In some embodiments, when the number of data processing tasks included in any task group is greater than a number threshold, the following processing may also be performed: based on the data processing type of the data processing tasks included in any task group, the data processing tasks included in any task group are divided to obtain multiple new task groups; based on the multiple new task groups, any task group is replaced.

[0135] Each new task group includes data processing tasks of the same type.

[0136] As an example, after grouping multiple data processing tasks based on a target grouping strategy to obtain multiple task groups, when the number of data processing tasks in any task group exceeds a certain threshold, it is necessary to further divide the data processing tasks in any task group based on the data processing type of the data processing tasks in that task group, and replace any task group with multiple new task groups obtained from the further division. The number threshold can be preset by engineers based on their engineering experience, or it can be predicted based on the corresponding neural network model. It is not limited here. The data processing types include: add, delete, modify, and query.

[0137] For example, with a threshold of 5, multiple data processing tasks are grouped into one task group containing 10 data processing tasks. Among them, 4 data processing tasks are of the "add" type, which means adding new data to the database; 3 data processing tasks are of the "delete" type, which means deleting existing data from the database; and 3 data processing tasks are of the "retrieve" type, which means retrieving data from the database. Based on the data processing type of each data processing task, the 10 data processing tasks in this task group are further divided into 3 new task groups. The first new task group contains 4 "add" type tasks, the second new task group contains 3 "delete" type tasks, and the third new task group contains 3 "retrieve" type tasks. The original task group is replaced by the above 3 new task groups.

[0138] In this embodiment of the application, task groups exceeding the quantity threshold are further divided according to data processing type, and the original task groups are replaced with new task groups. This enables refined management of data processing tasks, making the data processing types of data processing tasks within each new task group more consistent.

[0139] In step 103, multiple task groups are executed in parallel. During the execution of each task group, for each data processing task in the task group, based on the execution plan included in the data processing task, the data processing statements included in the data processing task are executed to obtain the actual execution data of the data processing task.

[0140] As an example, multiple task groups are executed in parallel based on multiple worker nodes, with different worker nodes corresponding to different task groups.

[0141] During the execution of each task group, when the target grouping strategy is to group by clustering, each data processing task in the task group is executed sequentially to obtain the actual execution data of each data processing task; when the target grouping strategy is to group by decoupling, each data processing task in the task group is executed in parallel to obtain the actual execution data of each data processing task.

[0142] In some embodiments, the "parallel execution of multiple task groups" in step 103 can be achieved by the following process: determining the resources required to execute each task group from the database and caching the resources to the worker node corresponding to each task group; and executing multiple task groups in parallel on multiple worker nodes based on the resources cached in multiple worker nodes.

[0143] Different task groups correspond to different work nodes.

[0144] As an example, the data resources required to execute each task group are determined from the database, and the data resources are cached in the corresponding worker node of each task group. Based on the resources cached in multiple worker nodes, multiple task groups are executed in parallel on multiple worker nodes.

[0145] See Figure 10 , Figure 10 This is a schematic diagram of the third principle of the training method for the artificial intelligence model provided in this application embodiment. The data resources required for executing task group 1003 are determined from the database 1005 and cached in the working node 1001 corresponding to task group 1003. Similarly, the data resources required for executing task group 1004 are determined from the database 1005 and cached in the working node 1002 corresponding to task group 1004. While executing task group 1003 on working node 1001, task group 1004 is executed on working node 1002.

[0146] In this embodiment of the application, by determining the resources required for each task group from the database and caching these resources on the corresponding working nodes, multiple task groups can be executed in parallel on multiple working nodes, which significantly improves the overall efficiency and response speed of multiple data processing tasks.

[0147] In some embodiments, when the target grouping strategy is a strategy of grouping by clustering, the step 103 of "for each data processing task in the task group, based on the execution plan included in the data processing task, execute the data processing statements included in the data processing task to obtain the actual execution data of the data processing task" can be implemented by the following process: based on the order of each data processing task in the task group, execute each data processing task sequentially; wherein, during the execution of each data processing task, based on each execution plan included in the data processing task, execute the data processing statements to obtain the execution data of each execution plan; and determine the execution data of at least one execution plan included in the data processing task as the actual execution data.

[0148] In step 1012, the data processing task is obtained by combining historical data processing statements and at least one corresponding candidate execution plan, wherein the at least one candidate execution plan includes the historical execution plan.

[0149] As an example, each data processing task is executed sequentially based on its order within the task group. The order of each data processing task within the task group can be determined based on its priority. The higher the priority of a data processing task, the earlier it is executed. The priority of a data processing task can be determined by a comprehensive evaluation based on the urgency of the task, resource requirements, dependencies, and business importance.

[0150] See Figure 11 , Figure 11 This is a schematic diagram of the fourth principle of the training method for the artificial intelligence model provided in this application embodiment. Based on the priority of data processing task 1102, data processing task 1103 and data processing task 1104, the execution order of the above three data processing tasks is determined, and data processing task 1102, data processing task 1103 and data processing task 1104 are executed sequentially in working node 1101.

[0151] During the execution of each data processing task, the following processing is performed for each data processing task in the task group: When any execution plan of the data processing task is not a historical execution plan, the data processing statement is executed based on the arbitrary execution plan to obtain the actual execution data of the arbitrary execution plan; when any execution plan of the data processing task is a historical execution plan, the historical execution data of the historical execution plan is obtained from the historical execution record, and the historical execution data is determined as the actual execution data of the arbitrary execution plan, and the actual execution data of all execution plans is determined as the actual execution data of the data processing task.

[0152] Perform the above processing on each data processing task in the task group to obtain the actual execution data for each data processing task.

[0153] In this embodiment, tasks are executed sequentially based on their order within the task group, and the method of obtaining execution data is flexibly handled according to the type of execution plan (historical execution plan or non-historical execution plan). For non-historical execution plans, data processing statements are executed directly to obtain actual execution data, ensuring the real-time performance and accuracy of data processing. For historical execution plans, historical execution data is obtained directly from historical execution records, avoiding redundant calculations, saving time and computing resources, and significantly improving the efficiency and accuracy of data processing.

[0154] In some embodiments, when the target grouping strategy is a strategy of grouping in a decoupling manner, the step 103 of "for each data processing task in the task group, based on the execution plan included in the data processing task, execute the data processing statements included in the data processing task to obtain the actual execution data of the data processing task" can be implemented by the following process: based on at least one data processing task included in the task group, execute at least one data processing task in parallel; wherein, during the execution of each data processing task, based on each execution plan included in the data processing task, execute data processing statements to obtain the execution data of each execution plan; and determine the execution data of at least one execution plan included in the data processing task as the actual execution data.

[0155] As an example, based on at least one data processing task included in the task group, at least one data processing task is executed in parallel. The specific processing for executing each data processing statement can be found above and will not be repeated here.

[0156] See Figure 12 , Figure 12 This is a fifth principle diagram of the training method for the artificial intelligence model provided in this application embodiment. The task group includes three data processing tasks: data processing task 1202, data processing task 1203 and data processing task 1204. The above three data processing tasks are executed in parallel in the working node 1201.

[0157] In this embodiment of the application, by executing data processing tasks in the task group in parallel, the concurrent computing capabilities of the working nodes are fully utilized, which significantly improves the execution efficiency and response speed of the tasks. At the same time, each data processing task can run independently, reducing the dependency and waiting time between tasks.

[0158] In step 104, the execution data of each data processing task is predicted using an artificial intelligence model to obtain the predicted execution data of each data processing task, and the artificial intelligence model is trained based on the actual execution data and the predicted execution data.

[0159] As an example, an artificial intelligence model is used to encode the data processing statements, at least one execution plan, and estimated data for each execution data for each data processing task. This results in the encoding features of the data processing statements, at least one execution plan, and estimated data for each execution data. Based on these encoding features, execution data prediction is performed for each data processing task to obtain the predicted execution data. The loss value is determined based on the difference between the actual execution data and the predicted execution data. By minimizing the loss value, the parameters of the artificial intelligence model are updated to obtain the trained artificial intelligence model.

[0160] Among them, the estimated data is the predicted cost information obtained by the optimizer in evaluating each execution plan before the data processing statement is executed. In the training process of artificial intelligence model, it can be used as a reference indicator or constraint to guide the training direction of artificial intelligence model.

[0161] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0162] In the field of data querying, with the widespread application of database systems and the explosive growth of data volume, users have increasingly higher requirements for query performance. As a key component of database management systems, the query optimizer is primarily responsible for analyzing and optimizing the data processing statements submitted by users. While optimizers in related technologies can automatically evaluate and select the optimal execution plan based on data processing statements and statistical data, there are instances where evaluation bias can lead to low actual execution performance. Artificial intelligence (AI)-based optimizers offer certain optimization enhancement capabilities, combining data processing statements with AI models to generate specific execution plans. Furthermore, since AI models are trained on actual execution data, they can effectively avoid the shortcomings of traditional optimizers. However, due to the high cost of acquiring execution data, this can prevent AI optimizers from intervening effectively in a timely manner under limited resources.

[0163] To address the problems existing in related technologies, this application proposes a training method for an artificial intelligence model, which can immediately allocate a large amount of resources to quickly interpolate execution data, thereby accelerating the entire training process of the AI ​​model (i.e., artificial intelligence model) and shortening the preparation time of the AI ​​optimizer. It also provides a complete implementation architecture for reference.

[0164] The main idea of ​​the AI ​​model training method proposed in this application is to apply for a sufficient number of pre-training instances during the AI ​​model training process and effectively divide the pre-training load (i.e., the multiple data processing tasks mentioned above) so that the divided pre-training load can be executed in parallel on the pre-training instances, thereby effectively reducing the time consumption of the pre-training stage. At the same time, the pre-training load is constructed by combining the production load and the AI ​​optimizer, and the pre-training tasks are executed to supplement the complete data used to train the AI ​​model. Then, the AI ​​model is trained by encoding the complete data and the encoding results based on the complete data to obtain the trained AI model. The AI ​​model training method proposed in this application fully leverages the characteristics of the large resource pool mode of the cloud environment, and can train the AI ​​optimizer to a near real-time intervention level, thereby meeting the needs of rapid automatic optimization of business.

[0165] The AI ​​model training method proposed in this application can accelerate pre-training by rapidly investing significant resources, thus shortening the AI ​​optimizer preparation time. Its main idea is to refine and expand the pre-training load based on production workloads, while simultaneously dividing the pre-training load to fully parallelize its execution across multiple pre-training instances. This AI model training method is suitable for business environments with optimization needs and the ability to quickly deliver database instances (e.g., public cloud, private cloud, and traditional deployment environments), where the business environment only needs to provide the necessary computing resources and data storage capabilities.

[0166] The training method for the artificial intelligence model proposed in this application will be described below from two aspects: the architecture of the training service and the integration of the training service.

[0167] 1. The architecture of the training service.

[0168] The design goal of the AI ​​optimization service (i.e., training service) is to create an enterprise-level, high-efficiency solution. Enterprise-level services not only need to be easy to integrate and interoperate, but also must possess high availability and high timeliness. The AI ​​model training service in this embodiment supports high-timeliness basic services and adopts a service-oriented architecture. Each service in this architecture is designed to be stateless, and the services agree on an interaction protocol, forming a three-layer architecture: See [link to relevant documentation] Figure 8 , Figure 8This is a schematic diagram of the first principle of the training method for the artificial intelligence model provided in this application embodiment. The data storage layer 801 is responsible for data storage, wherein the production load, pre-training load, complete data and AI model are all stored in the data storage layer 801; the training layer 802 is responsible for executing the pre-training load on the pre-training instances, wherein a pre-training cluster consists of multiple pre-training instances, each instance can have different specifications, and model training is completed in a dedicated graphics processing unit (GPU) cluster; the service layer 803 consists of a set of services, which is responsible for coordinating cluster resources, driving the training process, refining and expanding the load, and managing all metadata.

[0169] 1) Data storage layer 801

[0170] The data storage layer in this embodiment is used to support (store) the data of each training stage. The data storage layer adopts object storage service. Object storage has the advantages of high reliability, high scalability and high performance, and is the preferred storage service for the data storage layer. At the same time, the data storage layer can also use other storage services, such as distributed file systems or other storage services, as long as it is ensured that the relevant services and components can read and write the data therein.

[0171] The metadata contains descriptive information about the data at each stage; see [link to metadata]. Figure 8 Metadata is stored in metadata storage service 8031, which is a component of service layer 803. Metadata storage service 8031 ​​can use a highly available structured database (SQL database) or key-value database (KV database).

[0172] 2) Training layer 802

[0173] See Figure 8 Training layer 802 consists of pre-trained instances 8021, 8022, 8023, and 8024. Each group of pre-trained instances stores basic data in its cache. Each group of pre-trained instances supports one training job. Multiple groups of pre-trained instances are called a virtual cluster. Each pre-trained instance in the virtual cluster is called a worker node. A pre-trained instance can be a database instance. Figure 8 The pre-trained instances shown are merely illustrative examples, and the embodiments of this application do not limit the number of pre-trained instances in the service layer.

[0174] The pre-training instance is used only to analyze data processing statements (hereinafter referred to as statements) to obtain candidate execution plans for executing the statements (i.e., the data processing statements mentioned above), thereby obtaining estimated and execution data. Although the pre-training instance's cache stores basic data, this basic data is only used as an isolated read-only copy of the production data or randomly sampled data. If the database management service supports table-level cloning or backup recovery, the basic data stored in the pre-training instance's cache only needs to include the data tables involved in the statements in the pre-training load. Therefore, the pre-training instance does not need to have high availability. If the pre-training instance experiences a service failure during load replay, the system will automatically trigger a retry mechanism.

[0175] When executing statements, each statement can occupy a dedicated work node, and this work node is not shared with the execution tasks of other statements during the statement's execution, thus achieving a strong isolation mode. However, through analysis of the execution of pre-training tasks, it was found that a certain degree of concurrent execution does not negatively affect the performance of the work node. Therefore, independent statements can be executed simultaneously on the work node, thereby improving resource utilization.

[0176] The specifications (e.g., configuration and resource management) of pre-trained instances are not exposed as raw resources (e.g., CPU, memory, storage, etc.), but are defined and managed through an additional layer of specification abstraction. Pre-trained instances can be created, modified, and destroyed at any time without affecting the operation of production instances. When there are no pre-training tasks, all pre-trained instances can be released, or some can be reserved to improve the initial response speed of the training service.

[0177] When executing statements in a pre-trained instance, the execution plan is explicitly specified, and the entire optimization process is fully controlled. The pre-trained instance must have the ability to precisely intervene in the execution plan. At the same time, during the data collection phase, it is necessary to clearly distinguish between the overhead of the optimization process and the overhead of the actual execution.

[0178] 3) Service layer.

[0179] Compared to the isolation and temporariness of resources in the training layer, the service layer is shared by multiple users. Each service is always online and shared by multiple users. The services in the service layer have high availability and support elastic scaling capabilities (even if a single service node fails, the availability of the overall service will not be affected). In addition, since the training state is stored in metadata, the system will automatically retry even if the training task is interrupted due to a service node failure.

[0180] See Figure 8The training layer 803 includes metadata storage service 8031, authentication and access control service 8032, resource management service 8033, job management service 8034, load extraction service 8035, load analysis service 8036, and load replay service 8037. The core services of the service layer will be explained below.

[0181] All training requests must be submitted to the service layer to generate corresponding training jobs. The training request carries the production instance's identity (ID), the production load's time span, load extraction parameters (i.e., the filtering strategy mentioned above), training resource budget, and training strategy.

[0182] The job management service is responsible for maintaining the training job acceptance queue, the progress of training jobs, and the archiving of completed training jobs. Each training job must sequentially perform tasks such as load extraction, load analysis, resource request, load replay, data processing, and model training. The job management service drives the entire training job process and assigns the above tasks to different task managers (which can be represented as services) to complete the corresponding work. Each task manager reports its completion status to the job management service so that the job management service can drive the subsequent process.

[0183] The load extraction service obtains production load based on the production instance ID carried in the training request. It then filters the production load based on the time span and load extraction parameters to obtain the filtered production load. Among these, the database audit log (i.e., the production log of the database mentioned above) can provide a complete statement execution history, while some databases also provide performance history (e.g., Oracle AWR, i.e., the production log of the database mentioned above). These data can all be used as data sources for load extraction. However, when the production instance lacks necessary features (i.e., when the database audit log and performance history are missing), the load extraction service needs to sample from the production database in real time based on the production instance ID to obtain the production load.

[0184] Depending on the richness of the data source, the data source may contain a complete execution history or only a partial execution history. That is, it may contain various dimensions of indicators such as complete or partial execution plans, statistical data, estimated data and execution data. Complete data indicators can be directly used as execution information for production workloads after conversion, so as to save pre-training resources. Among them, execution information (i.e., the historical execution records mentioned above) includes statements (i.e., the historical data processing statements mentioned above), execution plans (i.e., the historical execution plans mentioned above), statistical data, estimated data and execution data (i.e., the historical execution data mentioned above) and various dimensions of indicators.

[0185] The load analysis service extracts statement summaries and data definitions from the statements and statistics in the execution information. It also constructs a candidate execution plan set for each statement and builds a pre-training load based on the statements and candidate execution plan sets. Furthermore, based on the statement summaries and data definitions, it determines the resource requirements for each statement. These resource requirements are used to divide the pre-training load into different subtasks (groups), ensuring that statements in different subtasks have no correlation in resource requirements (basic data). That is, pre-training loads corresponding to statements using the same basic data are grouped into the same subtask as much as possible. On this basis, if related hot queries cause too many statements to cluster in the same subtask, the subtask can be further split. At the same time, it ensures that multiple subtasks after further splitting are not scheduled to the same pre-training instance to avoid impacting performance data due to resource contention (unless the AI ​​model requires modeling contention). When constructing the pre-training load, it is also necessary to avoid executing the same statement in the same way.

[0186] The resource request service (resource management service) generates the optimal resource specifications and quantities based on the resource requirements and pricing models of the pre-trained load. For example, the memory requirement for executing a single statement can be represented as "the amount of data in the associated table * the memory resident rate * the cache adjustment factor", and the CPU requirement can be represented as "the statement parallelism * the utilization adjustment factor". The resource occupancy time can be estimated based on the statement execution time in the production load, and the estimation results are adjusted based on historical data. In other words, the resource specifications and quantities are transformed into solving a mixed-integer linear programming problem.

[0187] The resource request service requests the pre-training instance resources of the above-mentioned resource specifications and quantity from the database management service, and initializes the basic data cached in the pre-training instance. If the database management service provides backup and recovery and instance cloning capabilities, the resource request service can directly call the management service to complete the delivery of the pre-training instance. If the management service provides table-level operations, specific requirements for the basic data can also be passed in when requesting resources, and the requested pre-training instance and training task can be associated. If the basic data in the pre-training is a sample of production data, the corresponding data needs to be imported into the delivered pre-training instance.

[0188] The load replay service performs pre-training load partitioning by comparing the resource requirements of each task in the pre-training load (i.e., the data processing task mentioned above) with the basic data cached in the pre-training instance. Each task contains multiple statements and corresponding candidate execution plans. When a statement is executed in the pre-training instance, the statement is executed based on the execution method corresponding to the specified candidate execution plan. At the same time, estimated data and execution data, as well as other metrics, are collected during the execution of the statement. The data collected after load replay is used as the complete data for model training.

[0189] The model training service calls the training service of the machine learning platform. Based on the training resource budget and strategy, it trains the AI ​​model by encoding the complete data and the encoding results based on the complete data, and then stores the trained AI model in the data storage layer.

[0190] 2. Integration of training services.

[0191] AI models need to be integrated into production instances to be effective. This means that the systems running production instances need to be upgraded to versions that support AI models. In the training client systems, load managers monitor load changes in real time (e.g., a production instance experiencing a processing anomaly) and periodically or based on load changes, request the AI ​​optimizer to prepare and intervene.

[0192] See Figure 9 , Figure 9 This is a schematic diagram of the second principle of the training method for the artificial intelligence model provided in this application embodiment. During the application process, the artificial intelligence optimizer (AI optimizer) 901 receives the query statement and calls the model from the model repository 903 based on the model manager 902 to obtain the execution plan predicted by the AI ​​model. Then, it executes the execution plan predicted by the AI ​​model based on the executor 904 to obtain the result set. During the training process, the production instance can call the AI ​​model through the integrated training client 905, training service manager 906 and model manager 902. The load manager 907 detects load changes in real time and requests the artificial intelligence optimizer 901 to prepare and intervene based on the load change situation.

[0193] The Training Service Manager can also be deployed as a standalone training service to detect training requests initiated by production instances and write the execution plans predicted by the AI ​​model back to the production instances. This reduces the integration requirements of production instances, and the Training Service Manager or the training service only needs to maintain the training service address, authentication information, and service status. The Model Manager is used to maintain AI model data or the identity (ID) of AI models. When the Model Manager is used to maintain AI model IDs, when the AI ​​optimizer requests to load an AI model, the Model Manager reads the model data from the model repository based on the AI ​​model ID. This reduces the memory required for model management and only caches frequently used models.

[0194] Production instances need to have the ability to integrate AI models in order to effectively call AI models. Specifically, the AI ​​optimizer obtains the AI ​​model and its execution interface by calling the model manager, and obtains the prediction results of the AI ​​model through the execution interface. However, in special cases, if the AI ​​model is only used to calibrate statistical data and generate specific calibration values, the existing interface of the production instance can be called directly to write data. In this case, the system running the production instance does not need to perform any additional integration, which is suitable for systems that are inconvenient to upgrade. However, it should be noted that this is only applicable to a limited number of model types.

[0195] The training client and training service communicate via Representational State Transfer (REST) ​​or Remote Procedure Call (RPC). This communication can be synchronous or asynchronous. In synchronous communication, the training client directly writes the job results to the model manager. In asynchronous communication, after submitting the job, the training client records the job number and marks it as incomplete. It then periodically checks the job status until completion. The job result may be a model number or complete model data. When the job result is a model number, the model manager maintains a model proxy. When the AI ​​optimizer requests to load an AI model, the model manager reads the corresponding model data from the model repository. In this embodiment, recording model numbers reduces the memory usage of the model manager, caching only frequently used models. Furthermore, the records maintained in the model manager can also be updated via callback functions of asynchronous requests.

[0196] The training method for the artificial intelligence model provided in this application can effectively reduce the preparation time for AI optimizer intervention, break through the resource limitations of production instances, give full play to the advantages of the large resource pool in the cloud deployment environment, and provide a variety of optional integration modes to facilitate the integration of AI optimizer into the workflow, thereby achieving timely and automatic load optimization.

[0197] The following description continues to illustrate the exemplary structure of the training device 555 for the artificial intelligence model provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software modules in the training device 555 for the artificial intelligence model stored in the memory 550 may include:

[0198] The task extraction module 5551 is used to acquire multiple data processing tasks, wherein each data processing task includes a data processing statement for the database and at least one corresponding execution plan.

[0199] The task grouping module 5552 is used to group the plurality of data processing tasks to obtain a plurality of task groups, wherein each task group includes at least one data processing task.

[0200] The task execution module 5553 is used to execute the multiple task groups in parallel. During the execution of each task group, for each data processing task in the task group, based on the execution plan included in the data processing task, the data processing statements included in the data processing task are executed to obtain the actual execution data of the data processing task.

[0201] The model training module 5554 is used to predict the execution data of each data processing task through the artificial intelligence model, obtain the predicted execution data of each data processing task, and train the artificial intelligence model based on the actual execution data and the predicted execution data.

[0202] In some embodiments, the task grouping module 5552 is further configured to determine a target grouping strategy for grouping the plurality of data processing tasks from a plurality of grouping strategies; and to group the plurality of data processing tasks based on the target grouping strategy to obtain the plurality of task groups.

[0203] In some embodiments, the task grouping module 5552 is further configured to, when the target grouping strategy is a strategy of grouping by clustering, obtain the resources required to execute each data processing task; based on the resources required for each data processing task, cluster the plurality of data processing tasks to obtain a plurality of clusters, and determine the plurality of clusters as the plurality of task groups.

[0204] In some embodiments, the task grouping module 5552 is further configured to select multiple target resources from the resources required by the multiple data processing tasks, wherein the difference between the target resources is greater than a difference threshold; and to use the multiple target resources as cluster centers to cluster the multiple data processing tasks to obtain multiple clusters.

[0205] In some embodiments, the task execution module 5553 is further configured to execute each data processing task sequentially based on the order of each data processing task in the task group; wherein, during the execution of each data processing task, the data processing statement is executed based on each execution plan included in the data processing task to obtain execution data for each execution plan; and the execution data of at least one execution plan included in the data processing task is determined as the actual execution data.

[0206] In some embodiments, the task grouping module 5552 is further configured to, when the target grouping strategy is a strategy of grouping by decoupling, obtain the resources required to execute each data processing task; and, based on the resources required for each data processing task, group the plurality of data processing tasks according to resource coupling relationships to obtain the plurality of task groups.

[0207] In some embodiments, the task grouping module 5552 is further configured to, based on the resources required by each data processing task, determine from the plurality of data processing tasks the data processing tasks that have resource coupling relationships and the data processing tasks that do not have resource coupling relationships; decouple the data processing tasks that have resource coupling relationships to obtain decoupled data processing tasks, and divide the decoupled data processing tasks and the data processing tasks that do not have resource coupling relationships into corresponding task groups, wherein there are no resource coupling relationships between the data processing tasks included in each task group.

[0208] In some embodiments, the task execution module 5553 is further configured to execute at least one data processing task in parallel based on at least one data processing task included in the task group; wherein, during the execution of each data processing task, the data processing statement is executed based on each execution plan included in the data processing task to obtain execution data for each execution plan; and the execution data of at least one execution plan included in the data processing task is determined as the actual execution data.

[0209] In some embodiments, the task grouping module 5552 is further configured to divide the data processing tasks included in any task group based on the data processing type of the data processing tasks included in any task group to obtain multiple new task groups, wherein the data processing tasks included in each new group have the same data processing type; and replace any task group based on the multiple new task groups.

[0210] In some embodiments, the task execution module 5553 is further configured to determine the resources required to execute each task group from the database, and cache the resources to the working node corresponding to each task group, wherein different task groups correspond to different working nodes; and execute the multiple task groups in parallel on the multiple working nodes based on the resources cached in the multiple working nodes.

[0211] In some embodiments, the task extraction module 5551 is further configured to obtain at least one historical data processing statement corresponding to a production instance in the production environment of the database; generate at least one corresponding candidate execution plan for each historical data processing statement; and combine the historical data processing statement and the corresponding at least one candidate execution plan to obtain the data processing task.

[0212] In some embodiments, the task extraction module 5551 is further configured to: when historical execution records of the production instance are extracted from the production logs of the database, filter out at least one historical data processing statement from the historical execution records; when historical execution records of the production instance are not extracted from the production logs of the database, collect the execution records of the production instance in real time, and filter out at least one historical data processing statement from the collected execution records.

[0213] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. An electronic device's processor reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the electronic device to perform the artificial intelligence model training method described above in this application.

[0214] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the training method of the artificial intelligence model provided in this application.

[0215] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0216] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0217] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hypertext Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0218] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0219] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for training an artificial intelligence model, characterized in that, The method includes: Obtain multiple data processing tasks, wherein each data processing task includes a data processing statement for the database and at least one corresponding execution plan; The multiple data processing tasks are grouped to obtain multiple task groups, wherein each task group includes at least one data processing task. The multiple task groups are executed in parallel. During the execution of each task group, for each data processing task in the task group, based on the execution plan included in the data processing task, the data processing statements included in the data processing task are executed to obtain the actual execution data of the data processing task. The artificial intelligence model is used to predict the execution data of each data processing task to obtain the predicted execution data of each data processing task. Based on the actual execution data and the predicted execution data, the artificial intelligence model is trained.

2. The method according to claim 1, characterized in that, The process of grouping the multiple data processing tasks to obtain multiple task groups includes: From multiple grouping strategies, a target grouping strategy is determined to group the multiple data processing tasks. Based on the target grouping strategy, the multiple data processing tasks are grouped to obtain the multiple task groups.

3. The method according to claim 2, characterized in that, The process of grouping the multiple data processing tasks based on the target grouping strategy to obtain the multiple task groups includes: When the target grouping strategy is a strategy of grouping by clustering, obtain the resources required to execute each of the data processing tasks; Based on the resources required for each data processing task, the multiple data processing tasks are clustered to obtain multiple clusters, and the multiple clusters are determined as the multiple task groups.

4. The method according to claim 3, characterized in that, Based on the resources required by each data processing task, the multiple data processing tasks are clustered to obtain multiple clusters, including: Multiple target resources are selected from the resources required by the multiple data processing tasks, wherein the difference between the target resources is greater than a difference threshold; Using the multiple target resources as cluster centers, the multiple data processing tasks are clustered to obtain multiple clusters.

5. The method according to claim 3, characterized in that, For each data processing task in the task group, based on the execution plan included in the data processing task, the data processing statements included in the data processing task are executed to obtain the actual execution data of the data processing task, including: Each data processing task is executed sequentially based on its order within the task group. In the process of executing each of the data processing tasks, based on each of the execution plans included in the data processing task, the data processing statement is executed to obtain the execution data of each of the execution plans, and the execution data of at least one execution plan included in the data processing task is determined as the actual execution data.

6. The method according to claim 2, characterized in that, The process of grouping the multiple data processing tasks based on the target grouping strategy to obtain the multiple task groups includes: When the target grouping strategy is a grouping strategy performed through decoupling, the resources required to execute each data processing task are obtained; Based on the resources required for each data processing task, the multiple data processing tasks are grouped according to resource coupling relationships to obtain the multiple task groups.

7. The method according to claim 6, characterized in that, The data processing tasks are grouped based on resource coupling relationships according to the resources required for each data processing task, resulting in multiple task groups, including: Based on the resources required for each of the data processing tasks, determine the data processing tasks that have resource coupling relationships and the data processing tasks that do not have resource coupling relationships from the plurality of data processing tasks; The data processing tasks with resource coupling are decoupled to obtain decoupled data processing tasks. The decoupled data processing tasks and the data processing tasks without resource coupling are divided into corresponding task groups, wherein there is no resource coupling between the data processing tasks included in each task group.

8. The method according to claim 5, characterized in that, For each data processing task in the task group, the step of executing the data processing statements included in the data processing task based on the execution plan included in the data processing task to obtain the actual execution data of the data processing task includes: At least one of the data processing tasks included in the task group is executed in parallel. In the process of executing each of the data processing tasks, based on each of the execution plans included in the data processing task, the data processing statement is executed to obtain the execution data of each of the execution plans, and the execution data of at least one execution plan included in the data processing task is determined as the actual execution data.

9. The method according to any one of claims 1-8, characterized in that, When the number of data processing tasks included in any of the task groups exceeds a certain threshold, the method further includes: Based on the data processing type of the data processing tasks included in any of the task groups, the data processing tasks included in any of the task groups are divided to obtain multiple new task groups, wherein the data processing tasks included in each of the new groups have the same data processing type. Replace any of the aforementioned task groups based on the multiple new task groups.

10. The method according to any one of claims 1-8, characterized in that, The parallel execution of the multiple task groups includes: From the database, the resources required to execute each task group are determined, and the resources are cached in the working node corresponding to each task group, wherein different task groups correspond to different working nodes; Based on the resources cached in the multiple worker nodes, the multiple task groups are executed in parallel on the multiple worker nodes.

11. The method according to any one of claims 1-8, characterized in that, The acquisition of multiple data processing tasks includes: In the production environment of the database, obtain at least one historical data processing statement corresponding to the production instance; For each historical data processing statement, at least one corresponding candidate execution plan is generated, and the historical data processing statement and the corresponding at least one candidate execution plan are combined to obtain the data processing task.

12. The method according to claim 11, characterized in that, The statement for obtaining at least one historical data processing statement corresponding to the production instance includes: When historical execution records of the production instance are extracted from the production logs of the database, at least one historical data processing statement is selected from the historical execution records; When no historical execution record of the production instance is extracted from the production log of the database, the execution record of the production instance is collected in real time, and at least one historical data processing statement is selected from the collected execution record.

13. A training device for an artificial intelligence model, characterized in that, The device includes: The task extraction module is used to obtain multiple data processing tasks, wherein each data processing task includes a data processing statement for the database and at least one corresponding execution plan; The task grouping module is used to group the multiple data processing tasks to obtain multiple task groups, wherein each task group includes at least one data processing task. The task execution module is used to execute the multiple task groups in parallel. During the execution of each task group, for each data processing task in the task group, based on the execution plan included in the data processing task, the data processing statements included in the data processing task are executed to obtain the actual execution data of the data processing task. The model training module is used to predict the execution data of each data processing task through the artificial intelligence model, obtain the predicted execution data of each data processing task, and train the artificial intelligence model based on the actual execution data and the predicted execution data.

14. An electronic device, characterized in that, The electronic device includes: Memory is used to store computer programs or computer-executable instructions. A processor, when executing a computer program or computer-executable instructions stored in the memory, implements the training method of the artificial intelligence model according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The device stores a computer program or computer-executable instructions, which, when executed by a processor, implement the training method for the artificial intelligence model according to any one of claims 1 to 12.

16. A computer program product comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by a processor, they implement the training method of the artificial intelligence model according to any one of claims 1 to 12.

Images