Agent model-based reasoning query dynamic adaptive optimization method and device

Abstract

The invention relates to an agent model-based reasoning query dynamic adaptive optimization method and device. The method comprises the steps that a data query request is received, and the data query request comprises to-be-processed data and system resources; generating a plurality of candidate query plans according to the data query request, wherein each candidate query plan comprises a distribution state of to-be-processed data and a corresponding system resource configuration; inputting the plurality of candidate query plans into a deep learning model, and predicting to obtain execution time corresponding to each candidate query plan; and selecting the candidate query plan with the shortest execution time for execution. Through the method, reasoning query dynamic adaptive optimization can be realized, and the data query efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of big data query, and in particular to a method and device for dynamic adaptive optimization of reasoning query based on an agent model. Background Art

[0002] In recent years, with the rapid development of big data and artificial intelligence technologies, the scale of data has continued to grow, and the process of data analysis has become more and more complex, usually involving complex machine learning and deep learning technologies to mine knowledge from massive data. The data analysis process is reflected in the form of queries, and complex machine learning and deep learning models are encapsulated in user-defined functions (UDFs). On large-scale data, processing complex machine learning UDF queries is usually very time-consuming. For example, it usually takes several weeks to process a vehicle identification filter query on a day's surveillance video data. Related technologies propose a UDF filter query optimization method:

[0003] S1, define the problem to be solved; based on the correlation proxy model, first explain the search space for solving the problem; then define the reordering optimization problem for UDF filtering queries and find the optimal query execution plan; finally, prove that the optimization problem is an NP-complete problem;

[0004] S2, build the system framework; on the basis of the correlation proxy model, build the UDF filtering query processing system framework based on the rearrangement technology;

[0005] S3, design a solution algorithm; based on the correlation proxy model, design a solution algorithm for the reordering optimization problem of UDF filtering queries;

[0006] S4, design an optimization algorithm; according to the characteristics of the problem to be solved, further refine the underlying search tree structure to provide a fine-grained search tree; and propose an accuracy distribution algorithm during the search process to further narrow the upper and lower bounds of the execution cost of the candidate execution plan.

[0007] However, when data characteristics change or system resources fluctuate, the execution efficiency of the above data query is still low.

[0008] Currently, no effective solution has been proposed for the problem that data query methods in related technologies are not efficient enough. Summary of the invention

[0009] Based on this, it is necessary to provide a method, device and storage medium for dynamic adaptive optimization of reasoning queries based on an agent model that can improve execution efficiency in response to the above technical problems.

[0010] In a first aspect, the present application provides a method for dynamic adaptive optimization of reasoning queries based on a proxy model, comprising:

[0011] receiving a data query request, wherein the data query request includes data to be processed and system resources;

[0012] Generate multiple candidate query plans according to the data query request, each of the candidate query plans includes the distribution status of the data to be processed and the corresponding system resource configuration;

[0013] Inputting the multiple candidate query plans into a deep learning model to predict the execution time corresponding to each of the candidate query plans;

[0014] The candidate query plan with the shortest execution time is selected for execution.

[0015] In one embodiment, generating a plurality of candidate query plans according to the data query request includes:

[0016] Extract data at predicted distances from multiple data to be processed, so as to obtain multiple batches of data to be processed with different distribution states;

[0017] Different system resources are configured, and different batches of data to be processed are combined with different system resources to obtain multiple different candidate query plans.

[0018] In one embodiment, inputting the plurality of candidate query plans into a deep learning model comprises:

[0019] Encoding the candidate query plan and the current data to be processed;

[0020] The encoded candidate query plan and the current data to be processed are input into the deep learning model.

[0021] In one embodiment, the deep learning model includes: an input module, a fully connected module, and an output module connected in sequence; wherein,

[0022] The input module includes a convolution layer, a batch normalization layer, an activation function and a pooling layer;

[0023] The fully connected module includes a flattening layer, a fully connected layer, an activation function, and a random dropout layer;

[0024] The output module includes a fully connected layer.

[0025] In one embodiment, before inputting the plurality of candidate query plans into the deep learning model, the method further includes:

[0026] Acquire a training sample, wherein the training sample includes a plurality of query plans and data to be processed;

[0027] encoding the multiple query plans and the data to be processed;

[0028] The encoded multiple query plans and the data to be processed are input into the initial deep learning model for training.

[0029] In one embodiment, the encoded multiple query plans and the data to be processed are input into the initial deep learning model for training, including:

[0030] Predicting the execution time of each of the query plans;

[0031] Compare the predicted execution time of each query plan with the actual execution time of each query plan to obtain a prediction error;

[0032] Parameters of the deep learning model are adjusted based on the prediction error.

[0033] In one embodiment, after selecting the candidate query plan with the shortest execution time for execution, the method further includes:

[0034] During the execution of the candidate query plan, real-time monitoring of statistical information, the statistical information including a query plan selection rate or an available CPU number;

[0035] When the statistical information meets the preset conditions, the deep learning model is recalled to select a new round of query plans.

[0036] In one of the embodiments, the statistical information satisfies a preset condition including: the statistical information mutates and the mutation value exceeds a threshold.

[0037] In a second aspect, in this embodiment, a dynamic adaptive optimization device for reasoning queries based on a proxy model is provided, and the device includes:

[0038] A receiving module, used for receiving a data query request, wherein the data query request includes data to be processed and system resources;

[0039] A generating module, configured to generate a plurality of candidate query plans according to the data query request, each of the candidate query plans including a distribution state of the data to be processed and a corresponding system resource configuration;

[0040] A prediction module, used to input the multiple candidate query plans into a deep learning model to predict the execution time corresponding to each of the candidate query plans;

[0041] The execution module is used to select the candidate query plan with the shortest execution time for execution.

[0042] In a third aspect, in this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the agent model-based reasoning query dynamic adaptive optimization method described in the first aspect are implemented.

[0043] The above-mentioned agent-based model-based reasoning query dynamic adaptive optimization method, device and storage medium generate multiple candidate query plans, and each candidate query plan takes into account the distribution status of the data to be processed and the corresponding system resource configuration, uses a deep learning model to predict the execution time of the candidate query plan, and selects the candidate query plan with the shortest predicted execution time for execution. In this way, different query plans will be dynamically generated and selected during the entire query cycle to adapt to changes in data distribution and system resources. The query plan finally determined can be adjusted with the distribution status of the queried data and changes in system resources, avoiding the disadvantages of traditional data query methods that only plan once, and ensuring that the query plan is always close to optimal. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A hardware structure block diagram of a terminal of a method for dynamically adaptively optimizing reasoning queries based on an agent model in one embodiment;

[0045] Figure 2 A schematic diagram of a flow chart of a method for dynamic adaptive optimization of reasoning queries based on a proxy model in one embodiment;

[0046] Figure 3 A structural block diagram of a deep learning model in one embodiment;

[0047] Figure 4 A flowchart of a method for training a deep learning model in one embodiment;

[0048] Figure 5 It is a flowchart of a method for dynamic adaptive optimization of reasoning query based on a proxy model in another embodiment;

[0049] Figure 6 It is a structural block diagram of a dynamic adaptive optimization device for reasoning query based on an agent model in one embodiment;

[0050] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0053] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 1 is a hardware structure block diagram of a terminal of a method for dynamically adaptively optimizing reasoning queries based on an agent model according to an embodiment of the present application. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 101 and memory 102 for storing data, wherein processor 101 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 103 and an input and output device 104 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0054] The memory 102 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the agent model-based reasoning query dynamic adaptive optimization method in this embodiment. The processor 101 executes various functional applications and data processing by running the computer program stored in the memory 102, that is, to implement the above method. The memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 102 may further include a memory remotely arranged relative to the processor 101, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0055] The transmission device 103 is used to receive or send data via a network. The above network includes a wireless network provided by a communication provider of the terminal. In one example, the transmission device 103 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 103 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0056] Traditional query optimization methods can usually only optimize the execution of UDF filtering queries to a limited extent. For example, the predicate pushdown method can only push the query predicate down to the machine learning UDF that generates the corresponding label column. The machine learning UDF still needs to process all input data because the filter requires the machine learning UDF to generate the corresponding category label and select the corresponding category. The reordering technology finds the optimal order based on the execution cost (Cost) and selectivity (Selectivity) of the machine learning UDF, but the first UDF still needs to process all input data. In addition, pre-calculating and storing all the label columns generated by the machine learning UDF will cause huge computing and storage overhead; at the same time, since some label columns may not be used, it also causes a huge waste of computing and storage resources.

[0057] In order to optimize the execution of UDF filtering queries, the related art cascades two lightweight models before the time-consuming UDF: a lightweight customized model for the current query filter and a differential detection that evaluates the differences between different frames in the time-series video; thereby reducing the amount of data that needs to be processed by the subsequent time-consuming UDF and reducing the time required to process the entire query. Furthermore, based on the independence assumption, a lightweight PP is trained for each UDF filter in a more fine-grained manner, and these lightweight PPs are cascaded before the first UDF, further reducing the time required for the entire query processing. Furthermore, the related art proposes a method that can fully utilize the correlation between different UDF filters to further optimize the execution of complex UDF filtering queries.

[0058] The above UDF filtering query optimization methods are a "plan first, execute later" method, that is, only one fixed query plan is generated in the entire query cycle. Most of the time, the statistical information obtained by the system is very inaccurate. In the stream processing scenario, the characteristics of the data may change continuously. If only one fixed query plan is generated in the entire query cycle, it cannot adapt to the dynamic changes of data distribution and system resources. That is, the execution mode of planning once and then executing cannot be applied in all query scenarios.

[0059] Based on the above research and findings, in one embodiment, Figure 2 As shown in the figure, a dynamic adaptive optimization method for reasoning query based on agent model is provided. Figure 1 The terminal in is used as an example to illustrate, including the following steps:

[0060] Step S101: receiving a data query request, where the data query request includes data to be processed and system resources.

[0061] The data to be processed may refer to a data set that needs to be obtained through a query operation, and the data to be processed is stored in a database. System resources include, but are not limited to, hardware resources such as CPU, memory, disk I / O, and software resources such as database connections. When the system receives a data query request from a user, the request may contain specific data requirements, such as retrieving certain records from a database. Exemplarily, the user gives a data query request with the query "['TextTCLSNerTweebankNLPHF','TextTCLSNerDslimBaseHF']" and predicates "[""['B-PER','I-PER','B-LOC','B-MISC']"",""['I-ORG','B-ORG','B-LOC','B-PER','I-MISC','I-PER']""]", and determines the data to be processed and the system resources in the data query request.

[0062] Step S102, generating a plurality of candidate query plans according to the data query request, each candidate query plan including the distribution status of the data to be processed and the corresponding system resource configuration.

[0063] The system will generate a variety of different candidate query plans based on the received data query requests. That is, before the query starts, different data distributions and system resource states are preset, and multiple candidate query plans are generated based on these different data distributions and system resource states to cope with different data distributions and system resource states. Among them, each candidate query plan will take into account the different distribution of data, such as the changes in the proportion of each category in the data. And for each distribution state, the system resource configuration will also be considered, such as the number of CPU cores allocated to the query task, memory size, and CPU usage.

[0064] Step S103: input multiple candidate query plans into the deep learning model to predict the execution time corresponding to each candidate query plan.

[0065] A pre-trained deep learning model is used to predict the approximate running time of each candidate query plan to evaluate the performance of these plans. Optionally, the deep learning model can be a convolutional neural network (CNN) capable of processing one-dimensional input data, which can gradually extract features through a series of convolutional layers, batch normalization layers, activation functions and pooling layers, and finally regress through a fully connected layer.

[0066] Step S104: Select the candidate query plan with the shortest execution time for execution.

[0067] The system will select the candidate query plan with the shortest predicted execution time for actual execution to maximize query efficiency, reduce user waiting time, and effectively utilize system resources.

[0068] In the above steps S101 to S104, multiple candidate query plans are generated, and each candidate query plan takes into account the distribution status of the data to be processed and the corresponding system resource configuration, and the deep learning model is used to predict the execution time of the candidate query plan, and the candidate query plan with the shortest predicted execution time is selected for execution. In this way, different query plans will be dynamically generated and selected during the entire query cycle to adapt to changes in data distribution and system resources. The query plan finally determined can be adjusted with changes in the distribution status of the queried data and system resources, avoiding the disadvantages of traditional data query methods that only plan once, and ensuring that the query plan is always close to the optimal.

[0069] In one embodiment, when multiple candidate query plans are generated based on a data query request, data can be extracted from multiple data to be processed at predicted distance intervals to obtain multiple batches of data to be processed with different distribution states; different system resources are configured, and different batches of data to be processed and different system resources are combined to obtain multiple different candidate query plans.

[0070] In this embodiment, data can be extracted in batches at a certain interval from the data to be processed to obtain data in multiple different distribution states, such as extracting 5,000 data every 20,000 data from the first 100,000 data. And the resources that can be used by the system are limited. For example, the number of vCPUs (virtual central processing units) that can be used by the system can be limited to 12, 14, or 16. The combination of different batches of data and different system resources is used as an optimization condition to obtain multiple candidate query plans.

[0071] In one embodiment, when multiple candidate query plans are input into a deep learning model, the candidate query plans and the current data to be processed may be encoded first; and the encoded candidate query plans and the current data to be processed may be input into the deep learning model.

[0072] Candidate query plans and current data to be processed are stored in a specific structure, and deep learning models require numerical input. Therefore, before inputting candidate query plans and current data to be processed into the deep learning model, they need to be converted into a form that the model can understand and process, that is, converting them from structured information into numerical vectors or tensors, and then inputting them into the deep learning model.

[0073] In one embodiment, a deep learning model is provided, the deep learning model comprising: an input module, a fully connected module, and an output module connected in sequence. Each module is introduced below.

[0074] The input module mainly consists of four parts, each of which contains a convolution layer, a batch normalization layer, an activation function, and a pooling layer. The difference is that the structures of the second, third, and fourth parts are similar to the first part, but the output channels are gradually reduced. These decreasing output channels layer by layer allow the model to continuously extract more complex features while reducing the dimension of the feature map, thereby improving the generalization ability of the model. As an example, the first part of the input module includes:

[0075] For a one-dimensional convolutional layer, set the number of input channels to 1, the number of output channels to 256, the convolution kernel size to 5, and the padding value to 2, so that the output data length is the same as the input data length.

[0076] Batch normalization layer, used to perform batch normalization on 256 channels, speeding up training and making the model stable.

[0077] The activation function adopts the Parametric ReLU activation function, which is used to adaptively learn the parameters of the activation function to improve the expressiveness of the model.

[0078] The maximum pooling layer sets the pooling window size to 2, which is used to halve the length of the feature map (downsampling), reduce model parameters, and extract more representative features.

[0079] The structures of the second, third, and fourth parts are similar to those of the first part, but the number of output channels is gradually reduced. The number of output channels of the second part is reduced to 128, keeping the convolution kernel size and padding unchanged; the number of output channels of the third part is reduced to 64; and the number of output channels of the fourth part is reduced to 32.

[0080] The fully connected module includes a flattening layer, a fully connected layer, an activation function, and a random dropout layer. Each part is described as follows:

[0081] The flattening layer is used to flatten the three-dimensional feature map into a one-dimensional vector to prepare for the subsequent fully connected layer.

[0082] Fully connected layer, this layer is used to learn more advanced features. The number of input features is 32 (the number of output channels of the last convolutional layer) multiplied by 34 (assuming that the output length after multiple layers of convolution and pooling is 34, which needs to be calculated based on actual conditions), and the number of output features is 128.

[0083] The activation function adopts the Parametric ReLU activation function, which is used to adaptively learn the parameters of the activation function to improve the expressiveness of the model.

[0084] The random dropout layer is used to randomly discard half of the neurons to prevent overfitting and improve the generalization ability of the model.

[0085] The output module includes a fully connected layer with an input feature number of 128 and an output feature number of 1, which is a single continuous value output for regression tasks.

[0086] As an example, Figure 3 The structure of the deep learning model is given, such as Figure 3 As shown in the figure, the English terms in the figure are explained as follows:

[0087] Input: Input, accept input data.

[0088] Max-Pool: Maximum pooling layer, downsampling, reducing the spatial size of the input by taking the maximum value of the local area while retaining the most important features.

[0089] BatchNorm: Batch normalization layer, which speeds up training and improves generalization by standardizing the input of each layer.

[0090] Conv: Convolutional layer, responsible for extracting features from the input data. Conv1-64 means that the first convolutional layer has 64 convolution kernels.

[0091] FC (Fully Connected Layer): Fully connected layer, all input neurons in this layer are fully connected to all output neurons.

[0092] Flatten: Flattening layer, which flattens a multidimensional array into a one-dimensional array to facilitate subsequent fully connected layer processing.

[0093] Output: Output layer, used to output predicted values.

[0094] PReLU (Parametric Rectified Linear Unit): The activation function in the residual network is similar to the ReLU activation function, which has a learnable parameter α, allowing the negative part to be not completely flat but with a slope of α.

[0095] The deep learning model provided in this embodiment is used to process one-dimensional sequence data and perform regression prediction. Among them, features are extracted through multiple convolutional layers, the batch normalization layer stabilizes the training process, PReLU is used to improve the nonlinear expression ability, the maximum pooling layer and the flattening layer reduce the data dimension, and finally the final regression prediction is performed through the fully connected layer.

[0096] In one embodiment, Figure 4 A flowchart of the training method of the deep learning model is given, such as Figure 4 As shown, the process includes the following steps:

[0097] Step S201, obtain training samples, the training samples include multiple query plans and data to be processed. The query plan can come from the actual database operation log, each query plan includes the distribution status of the data to be processed and the corresponding system resource configuration. The data to be processed can be a data set to be queried in the database.

[0098] Step S202: Encode multiple query plans and data to be processed. Convert candidate query plans and data to be processed from structured information into numerical vectors or tensors to conform to the input format of the deep learning model.

[0099] Step S203: Input the encoded multiple query plans and the data to be processed into the initial deep learning model for training. Input the encoded numerical vector or tensor into the deep learning model for training.

[0100] Step S204: predicting the execution time of each query plan. The output of the deep learning model is the execution time of the query plan.

[0101] Step S205, compare the predicted execution time of each query plan with the actual execution time of each query plan to obtain a prediction error. In order to evaluate the accuracy of the model, the execution time predicted by the model is compared with the execution time recorded when the query is actually executed to obtain an error value. The error value can be calculated by calculating the mean square error (MSE), the root mean square error (RMSE) or the mean absolute error (MAE).

[0102] Step S206, adjusting the parameters of the deep learning model according to the prediction error. By continuously adjusting the model parameters, the prediction capability can be gradually improved, so that the predicted execution time is closer to the actual execution time.

[0103] During the execution of the plan, some indicators will change, such as the selectivity or the number of available CPUs. If you continue to select the candidate query plan before the indicator changes, the candidate query plan may not match the current data distribution status or system resources, and the efficiency of query execution cannot be guaranteed.

[0104] In one embodiment, to solve the above problem, in the process of executing the candidate query plan, the statistical information is monitored in real time, and the statistical information includes the query plan selection rate or the number of available CPUs; when the statistical information meets the preset conditions, the deep learning model is re-called to select a new round of query plans. Among them, the statistical information meets the preset conditions including: the statistical information mutates, and the mutation value exceeds the threshold. In this embodiment, the relevant statistical information is monitored in real time. If it is monitored that the selection rate or the number of available CPUs and other information mutates, and the mutation value exceeds the threshold, the deep learning model is re-called to select a new round of plans to ensure the optimality of the query execution efficiency.

[0105] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0106] In one embodiment, Figure 5 A flowchart of a dynamic adaptive optimization method for reasoning queries based on a proxy model is provided. Figure 5 As shown, the process includes the following steps:

[0107] Step S301, receiving a data query request, the data query request includes data to be processed and system resources;

[0108] Step S302, generating multiple candidate query plans according to the data query request, each candidate query plan includes the distribution status of the data to be processed and the corresponding system resource configuration. Specifically, extracting data at intervals of predicted distances from the multiple data to be processed to obtain multiple batches of data to be processed with different distribution statuses;

[0109] Different system resources are configured, and different batches of data to be processed are combined with different system resources to obtain multiple different candidate query plans.

[0110] Step S303: Input multiple candidate query plans into the deep learning model to predict the execution time corresponding to each candidate query plan. Specifically, the candidate query plan and the current data to be processed are encoded; the encoded candidate query plan and the current data to be processed are input into the deep learning model.

[0111] Step S304: Select the candidate query plan with the shortest execution time for execution.

[0112] Step S305, during the execution of the candidate query plan, real-time monitoring of statistical information, the statistical information including the query plan selection rate or the number of available CPUs;

[0113] Step S306: if the statistical information meets the preset conditions, the deep learning model is re-called to select a new round of query plans. The statistical information meets the preset conditions including: the statistical information mutates and the mutation value exceeds a threshold.

[0114] In this embodiment, a deep learning model is used to perform dynamic and adaptive query plan selection. The execution time is predicted by a deep learning model, and the query plan is dynamically adjusted according to the real-time monitored data during the execution process, thereby ensuring the efficiency and robustness of the query execution. Specifically, there are multiple query plans throughout the query cycle, and the selection of the plan is adjusted as factors such as the distribution of the data involved in the query and system resources (such as CPU occupancy) change. The plan is always close to optimal. Diversified query plans: Different query plans are dynamically generated and selected throughout the query cycle to adapt to changes in data distribution and system resources. Dynamic adaptive optimization: Through real-time monitoring and adjustment, the query plan is ensured to be always close to optimal, avoiding the shortcomings of a single and unadjustable plan in traditional methods. Improve query efficiency: Through deep learning model prediction and real-time optimization, the execution efficiency of large-scale data queries is significantly improved.

[0115] Based on the same inventive concept, the embodiment of the present application also provides a proxy model-based reasoning query dynamic adaptive optimization device for implementing the proxy model-based reasoning query dynamic adaptive optimization method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more proxy model-based reasoning query dynamic adaptive optimization device embodiments provided below can refer to the limitations on the data query method above, and will not be repeated here.

[0116] In one embodiment, Figure 6 A dynamic adaptive optimization device for reasoning queries based on an agent model is provided. Figure 6 As shown, the device comprises:

[0117] The receiving module 201 is used to receive a data query request, where the data query request includes data to be processed and system resources;

[0118] A generating module 202, configured to generate a plurality of candidate query plans according to a data query request, each candidate query plan including a distribution state of data to be processed and a corresponding system resource configuration;

[0119] A prediction module 203 is used to input multiple candidate query plans into the deep learning model to predict the execution time corresponding to each candidate query plan;

[0120] The execution module 204 is used to select a candidate query plan with the shortest execution time for execution.

[0121] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0122] Each module in the above-mentioned agent model-based reasoning query dynamic adaptive optimization device can be implemented in whole or in part by software, hardware, and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0123] In addition, in combination with the inference query dynamic adaptive optimization method based on the proxy model provided in the above embodiment, a storage medium can also be provided in this embodiment to implement. The storage medium stores a computer program; when the computer program is executed by the processor, the steps of any one of the inference query dynamic adaptive optimization methods based on the proxy model in the above embodiment are implemented.

[0124] S1. In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0125] S2, receiving a data query request, the data query request includes the data to be processed and system resources;

[0126] S3, generating multiple candidate query plans according to the data query request, each candidate query plan including the distribution status of the data to be processed and the corresponding system resource configuration;

[0127] S4, inputting multiple candidate query plans into the deep learning model to predict the execution time corresponding to each candidate query plan;

[0128] S5, select the candidate query plan with the shortest execution time for execution.

[0129] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data to be processed. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a dynamic adaptive optimization method for reasoning queries based on an agent model is implemented.

[0130] Those skilled in the art will understand that Figure 7The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0131] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0132] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0133] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0134] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A dynamic adaptive optimization method for reasoning queries based on a proxy model, characterized in that: include: receiving a data query request, wherein the data query request includes data to be processed and system resources; Generate multiple candidate query plans according to the data query request, each of the candidate query plans includes the distribution status of the data to be processed and the corresponding system resource configuration; Inputting the multiple candidate query plans into a deep learning model to predict the execution time corresponding to each of the candidate query plans; The candidate query plan with the shortest execution time is selected for execution.

2. The method for dynamic adaptive optimization of reasoning query based on agent model according to claim 1, characterized in that: Generate multiple candidate query plans according to the data query request, including: Extract data at predicted distances from multiple data to be processed, so as to obtain multiple batches of data to be processed with different distribution states; Different system resources are configured, and different batches of data to be processed are combined with different system resources to obtain multiple different candidate query plans.

3. The method for dynamic adaptive optimization of reasoning query based on agent model according to claim 1, characterized in that: Inputting the plurality of candidate query plans into a deep learning model comprises: Encoding the candidate query plan and the current data to be processed; The encoded candidate query plan and the current data to be processed are input into the deep learning model.

4. The method for dynamic adaptive optimization of reasoning query based on agent model according to claim 1, characterized in that: The deep learning model includes: an input module, a fully connected module, and an output module connected in sequence; wherein, The input module includes a convolution layer, a batch normalization layer, an activation function and a pooling layer; The fully connected module includes a flattening layer, a fully connected layer, an activation function, and a random dropout layer; The output module includes a fully connected layer.

5. The method for dynamic adaptive optimization of reasoning query based on agent model according to claim 1, characterized in that: Before inputting the plurality of candidate query plans into the deep learning model, the method further includes: Acquire a training sample, wherein the training sample includes a plurality of query plans and data to be processed; encoding the multiple query plans and the data to be processed; The encoded multiple query plans and the data to be processed are input into the initial deep learning model for training.

6. The method for dynamic adaptive optimization of reasoning query based on agent model according to claim 5, characterized in that: Input the encoded query plans and the data to be processed into the initial deep learning model for training, including: Predicting the execution time of each of the query plans; Compare the predicted execution time of each query plan with the actual execution time of each query plan to obtain a prediction error; Parameters of the deep learning model are adjusted based on the prediction error.

7. The method for dynamic adaptive optimization of reasoning query based on agent model according to claim 1, characterized in that: After selecting the candidate query plan with the shortest execution time for execution, the method further includes: During the execution of the candidate query plan, real-time monitoring of statistical information, the statistical information including a query plan selection rate or an available CPU number; When the statistical information meets the preset conditions, the deep learning model is recalled to select a new round of query plans.

8. The method for dynamic adaptive optimization of reasoning query based on agent model according to claim 7, characterized in that: The statistical information meets the preset conditions including: The statistical information mutates, and the mutation value exceeds a threshold.

9. A dynamic adaptive optimization device for reasoning query based on agent model, characterized in that: The device comprises: A receiving module, used for receiving a data query request, wherein the data query request includes data to be processed and system resources; A generating module, configured to generate a plurality of candidate query plans according to the data query request, each of the candidate query plans including a distribution state of the data to be processed and a corresponding system resource configuration; A prediction module, used to input the multiple candidate query plans into a deep learning model to predict the execution time corresponding to each of the candidate query plans; The execution module is used to select the candidate query plan with the shortest execution time for execution.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the agent model-based reasoning query dynamic adaptive optimization method according to any one of claims 1 to 8 are implemented.

Images