Data coding-based reasoning query method and device containing proxy model

Abstract

The invention relates to a data coding-based reasoning query method and device containing a proxy model. The method comprises the following steps: obtaining query data, wherein the query data comprises a query text and query function data; determining a text code based on the query text, and determining a query parameter code based on the query function data; determining a query data code based on the text code and the query parameter code; the query data codes are input into a query time prediction model, query prediction time is output, a query result is determined on the basis of the query prediction time and a preset query plan algorithm, and the query time prediction model is obtained by training a plurality of query time prediction samples; the query time prediction sample is determined based on a preset query plan algorithm and historical query data. The query efficiency can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and device for reasoning query containing an agent model based on data coding. Background Art

[0002] In recent years, with the rapid development of big data and artificial intelligence technology, the scale of data has continued to grow, and the process of data analysis has become more and more complex. The data analysis process usually involves complex machine learning and deep learning technologies to mine knowledge from massive data. For example, based on the data collected by road surveillance cameras, scenarios such as traffic flow monitoring, road planning, and public transportation deployment usually involve a large amount of multi-source road surveillance video data, as well as complex neural network models for identifying different motor vehicles, non-motor vehicles, and pedestrians. The data analysis process is usually executed in the form of queries, and complex machine learning models and deep learning models are encapsulated in user-defined functions (UDFs).

[0003] However, in traditional large-scale data query technology, processing complex machine learning model UDFs queries is usually time-consuming. For example, it usually takes several weeks to process a vehicle identification filter query on a day's surveillance video data. Therefore, how to optimize the execution process of complex model user-defined function UDFs filter queries has attracted widespread attention in the academic community.

[0004] In traditional technologies, in the optimized complex machine learning model UDFs query algorithm, multiple executable query plans are usually determined based on the query data provided by the user and the custom function UDFs, and the execution time of each executable query plan is predicted, and the optimal query plan is determined accordingly.

[0005] However, in traditional technologies, the algorithm parameters involved in query algorithms, such as the time cost parameter for UDFs to process each piece of data, are usually used as fixed values ​​for calculation. However, algorithm parameters may be affected by various factors such as the distribution of query data and the correlation between UDFs, and may actually fluctuate, which reduces the accuracy of the algorithm's prediction of the execution time required for the execution plan, and further causes the optimal query plan to still take a long time, which means that the data query takes a long time and the query efficiency is low. Summary of the invention

[0006] Based on this, it is necessary to provide a reasoning query method and device based on data encoding and containing a proxy model that can improve data query efficiency in response to the above technical problems.

[0007] In a first aspect, the present application provides a reasoning query method containing a proxy model based on data encoding. The method comprises:

[0008] Acquire query data, wherein the query data includes query text and query function data;

[0009] determining a text encoding based on the query text, and determining a query parameter encoding based on the query function data;

[0010] Determine the query data code based on the text code and the query parameter code;

[0011] The query data encoding is input into a query time prediction model, the query prediction time is output, and the query result is determined based on the query prediction time and a preset query plan algorithm, wherein the query time prediction model is trained based on multiple query time prediction samples, and the query time prediction samples are determined based on the preset query plan algorithm and historical query data.

[0012] In one embodiment, before obtaining the query data, the method further includes:

[0013] Acquire the historical query data, and determine at least one data query plan based on the historical query data and the preset query plan algorithm;

[0014] Execute the at least one data query plan and determine the execution time corresponding to each data query plan;

[0015] The query time prediction sample is determined based on the historical query data, the execution time, and the preset query plan algorithm.

[0016] In one embodiment, the historical query data includes historical query text and historical query function data, and determining the query time prediction sample based on the historical query data, the execution time, and the preset query plan algorithm includes:

[0017] Determine query plan parameters based on the historical query data and the preset query plan algorithm, wherein the query plan parameters match the data query plan;

[0018] Determine a sample text code based on the historical query text, and determine a sample query parameter code based on the historical query function data and the query plan parameters;

[0019] Determine a sample query data code based on the sample text code and the sample query parameter code;

[0020] The query time prediction sample is determined based on the sample query data encoding and the execution time.

[0021] In one embodiment, the historical query text includes a plurality of historical text data, and determining the sample text encoding based on the historical query text includes:

[0022] Obtaining a preset number of to-be-encoded texts from the plurality of historical text data, wherein the semantic order of the preset number of to-be-encoded texts is consistent with the semantic order of the plurality of historical text data in the historical query text;

[0023] splicing the preset number of to-be-encoded texts based on the semantic order to obtain a to-be-encoded text segment;

[0024] The text segment to be encoded is input into a preset text embedding model to generate a sample embedding vector as the sample text encoding.

[0025] In one embodiment, the historical query function data includes a historical custom function and a historical custom query type parameter, and the determining of the sample query parameter code based on the historical query function data and the query plan parameter includes:

[0026] Based on the one-hot encoding rule, the historical custom function, the historical custom query type parameter, and the query plan parameter are respectively encoded and concatenated based on a first preset concatenation order to obtain the sample query parameter encoding.

[0027] In one embodiment, determining the sample query data code based on the sample text code and the sample query parameter code includes:

[0028] Based on a second preset concatenation order, the sample text code and the sample query parameter code are concatenated to obtain the sample query data code.

[0029] In one of the embodiments, after determining the query time prediction sample based on the sample query data encoding and the execution time, the method further includes:

[0030] Constructing a query time prediction model, wherein the query time prediction model is provided with training parameters;

[0031] Inputting the sample query data encoding into the query time prediction model to generate a query time prediction result;

[0032] Based on the difference between the query time prediction result and the execution time, the training parameters are iteratively adjusted until the difference meets a preset requirement.

[0033] In one embodiment, the query function data includes a custom function and a custom query type parameter, and the determining of the text encoding based on the query text and the determining of the query parameter encoding based on the query function data include:

[0034] Inputting the query text into a preset text embedding model to generate a query embedding vector as the text encoding;

[0035] Based on the one-hot encoding rule, the user-defined function and the user-defined query type parameters are encoded respectively, and then spliced ​​based on a third preset splicing order to obtain the query parameter encoding.

[0036] In one embodiment, obtaining the query data code based on the text code and the query parameter code includes:

[0037] Based on a fourth preset concatenation order, the text code and the query parameter code are concatenated to obtain the query data code.

[0038] In a second aspect, the present application also provides a reasoning query device containing a proxy model based on data encoding. The device comprises:

[0039] A query data acquisition module, used to acquire query data, wherein the query data includes query text and query function data;

[0040] An encoding module, used to determine a text encoding based on the query text, and to determine a query parameter encoding based on the query function data;

[0041] A query data code determination module, used to determine the query data code based on the text code and the query parameter code;

[0042] A query module is used to encode the query data and input it into a query time prediction model, output the query prediction time, and determine the query result based on the query prediction time and a preset query plan algorithm, wherein the query time prediction model is trained based on multiple query time prediction samples, and the query time prediction samples are determined based on the preset query plan algorithm and historical query data.

[0043] In a third aspect, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any of the above-mentioned data-encoded reasoning query methods containing a proxy model are implemented.

[0044] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-mentioned data-encoded reasoning query methods containing a proxy model in the first aspect.

[0045] The reasoning query method containing a proxy model based on data coding provided by the embodiment of the present application encodes the query text and the query function separately, and determines the query data coding after splicing, which can at least reflect the query data distribution law in the query data coding, and can also reflect the correlation between the query functions if there are multiple query functions. The query data coding is then input into the query time prediction model, and the query prediction time is output, so that the prediction of the query time is more accurate. The query result is then determined based on the query prediction time and the preset query plan algorithm, so that the optimal data query plan can be determined more effectively, the data query time can be reduced, and the query efficiency can be effectively improved.

[0046] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0048] Figure 1 An application environment diagram of a reasoning query method containing a proxy model based on data encoding in one embodiment;

[0049] Figure 2 A flowchart of a reasoning query method containing a proxy model based on data encoding in one embodiment;

[0050] Figure 3 A schematic diagram of a process for determining a query time prediction sample in one embodiment;

[0051] Figure 4 It is a structural block diagram of a reasoning query device containing a proxy model based on data encoding in one embodiment;

[0052] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] The terms "module", "unit", etc. used below are a combination of software and / or hardware that can implement a predetermined function. Although the devices described in the following embodiments are preferably implemented in hardware, the implementation of software, or a combination of software and hardware is also possible and conceivable.

[0055] In traditional technologies, query optimization methods for the execution process of filtering queries containing complex model user-defined functions (UDFs) can usually only optimize the execution of UDFs filtering queries to a limited extent. For example, the predicate pushdown method can only push the query predicate down to the machine learning UDFs that generate the corresponding label columns. The machine learning UDFs still need to process all input data because the filter requires the machine learning UDFs to generate the corresponding category labels and thus select the corresponding categories. The reordering technology finds the optimal execution order based on the execution cost (Cost) and selectivity (Selectivity) of the machine learning UDFs, but the first UDF still needs to process all input data. In addition, pre-calculating and storing all the label columns generated by the machine learning UDFs will cause huge computing and storage overhead. On the other hand, since some label columns may not be used, it will also cause a huge waste of computing and storage resources.

[0056] In recent years, in order to optimize the execution of UDFs filtering queries, NoScope (a video object detection system) has been proposed in traditional technology. It cascades two lightweight models before time-consuming UDFs, a lightweight customized model for the current query filter and a differential detection that evaluates the differences between different frames in the time-series video, thereby reducing the amount of data that needs to be processed by subsequent time-consuming UDFs, thereby reducing the time required to process the entire query. Traditional technology also proposes PP, which, based on the independence assumption, trains a lightweight PP for each UDF filter in a more fine-grained manner, and cascades these lightweight PPs before the first UDF, further reducing the time required for the entire query processing.

[0057] It is worth noting that the above methods ignore the correlation between different UDFs filters in the query, resulting in a longer query processing time. In the traditional technology, a query plan algorithm is proposed that can fully utilize the correlation between different UDFs filters and further optimize the execution of complex UDF filtering queries. The query plan algorithm mainly estimates the query execution time through the following steps and determines the optimal query plan based on this.

[0058] Step 1: UDF filter query q contains user-defined functions UDFs and their corresponding filters {σ1,…σ n}. A proxy model and its corresponding UDF and filter σ i The execution cost function is shown in formula (1):

[0059]

[0060] In formula (1), α i Is a proxy model The accuracy parameter, s i is the filter σ i The selection rate, Is a proxy model The time cost of processing each piece of data, r i Is a proxy model The screening rate, c i is a UDF The time cost of processing each piece of data. It is a proxy model generated based on the UDF given by the user after the user gives a UDF containing an ML (machine learning) model.

[0061] Step 2: Provide the target accuracy of the query The relationship between the added proxy model accuracy parameter α must satisfy formula (2):

[0062]

[0063] In formula (2), is the filter σ i The previous filter {σ1,…,σ i-1} selection rate under the condition.

[0064] Step 3: The execution cost function of the UDF filtering query based on the proxy model is formula (3):

[0065]

[0066] The execution time of a single proxy model and its corresponding UDF and filter can be calculated based on the cost function.

[0067] However, in the above query plan algorithm, on the one hand, it is necessary to obtain α in advance i ,s i , r i, c i For time-consuming and complex machine learning UDFs, it may be expensive to obtain this information. i , c i Once fixed, parameters such as time and time will not change. However, due to factors such as data distribution, correlation between UDFs, and system resources (such as CPU usage), these parameters will actually fluctuate on different data. This will reduce the accuracy of the estimated query execution time results, and the optimal query plan may still take a long time, which means that the data query time is longer and the query efficiency is lower.

[0068] Based on this, the present application provides a reasoning query method containing a proxy model based on data encoding that can improve the accuracy of data query prediction. The reasoning query method containing a proxy model based on data encoding can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 obtains the query data and sends it to the server 104, and the query data includes the query text and the query function data. The server 104 determines the text encoding based on the query text, and determines the query parameter encoding based on the query function data; determines the query data encoding based on the text encoding and the query parameter encoding; inputs the query data encoding into the query time prediction model, outputs the query prediction time, determines the query result based on the query prediction time and the preset query plan algorithm, and sends it to the terminal 102. Among them, the query time prediction model is obtained based on the training of multiple query time prediction samples, and the query time prediction samples are determined based on the preset query plan algorithm and historical query data. The above steps can also be performed by the terminal 102 or the server 104 alone, and this application does not limit this. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices, portable wearable devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0069] In one embodiment, Figure 2 As shown in FIG, a method for reasoning query based on data encoding and containing a proxy model is provided. Figure 1 The application scenario in is taken as an example to illustrate, including the following steps:

[0070] S201: Acquire query data, where the query data includes query text and query function data.

[0071] In an embodiment of the present application, the query data may include query text and query function data. Among them, the query text may include the text data to be processed by the query, and specifically, the text data may include at least one query statement, or a query paragraph composed of multiple query statements. The query function data may include a custom function and a custom query type parameter. Specifically, a custom function (UDF, User-Defined Function) may contain a machine learning model, which is used to specify the machine learning model for querying during the query process. The custom query type parameter may include a query predicate, which is used to specify the query filter condition, that is, the user customizes the type of the query result.

[0072] In some specific embodiments, the query text may include "['TextTCLSNerTweebankNLPHF', 'TextTCLSNerDslimBaseHF']", and the query predicate may include ""['I-ORG', 'B-ORG', 'B-LOC', 'B-PER', 'I-MISC', 'I-PER']""]". In other embodiments, the query data may also include data required for query processing and system resources required for query, etc.

[0073] S203: Determine a text encoding based on the query text, and determine a query parameter encoding based on the query function data.

[0074] In an embodiment of the present application, a text encoding can be determined based on the query text. Specifically, the query text can be input into an embedding vector model to determine an embedding vector as the text encoding. In other embodiments, the query text can also be input into a sentence embedding model, and the query text can be encoded into a vector of a fixed length as the text encoding after capturing the overall semantics of the query text. In other embodiments, the text encoding can also be determined based on the query text according to the word vector averaging method.

[0075] In an embodiment of the present application, the query parameter encoding can be determined based on the query function data. Specifically, the query function data can be encoded based on the one-hot encoding rule to determine the query parameter encoding. In other embodiments, the query function data can also be encoded based on the hash encoding rule to determine the query parameter encoding. In other embodiments, the features of the query function data can also be mapped to a continuous low-dimensional vector space through an embedding layer through a word embedding encoding model to determine the query parameter encoding.

[0076] In some embodiments, the query function data includes a custom function and a custom query type parameter, determining the text encoding based on the query text, and determining the query parameter encoding based on the query function data includes:

[0077] S301: Input the query text into a preset text embedding model to generate a query embedding vector as the text encoding.

[0078] S303: Based on the one-hot encoding rule, the user-defined function and the user-defined query type parameters are encoded respectively, and then concatenated based on a third preset concatenation order to obtain a query parameter code.

[0079] In the embodiment of the present application, the query text can be input into a preset text embedding model to generate a query embedding vector as the text encoding. In some specific embodiments, the preset text embedding model can be nomic-embed-text-v1.5.

[0080] Based on the one-hot encoding rule, after encoding the custom function and the custom query type parameter respectively, they are concatenated based on a third preset concatenation order to obtain the query parameter encoding. The third preset concatenation order may include a concatenation order of a custom function-custom query type parameter, or a concatenation order of a custom query type parameter-custom function.

[0081] It should be noted that the method of determining the text encoding and query parameter encoding during the query process should be consistent with the method of determining the sample text encoding and sample query parameter encoding during query time prediction model training.

[0082] S205: Determine the query data code based on the text code and the query parameter code.

[0083] In some embodiments, obtaining the query data encoding based on the text encoding and the query parameter encoding includes:

[0084] S401: Based on a fourth preset concatenation order, concatenate the text code and the query parameter code to obtain a query data code.

[0085] After determining the text code and the query parameter code, the text code and the query parameter code may be concatenated based on a fourth preset concatenation order. Specifically, the fourth concatenation order may include a concatenation order of text code-query parameter code, or a concatenation order of query parameter code-text code.

[0086] It should be noted that the third preset concatenation order and the fourth preset concatenation order that may be involved in the query process should be consistent with the corresponding concatenation order during the query time prediction model training.

[0087] S207: Encode the query data and input it into the query time prediction model, output the query prediction time, and determine the query result based on the query prediction time and a preset query plan algorithm, wherein the query time prediction model is trained based on multiple query time prediction samples, and the query time prediction samples are determined based on a preset query plan algorithm and historical query data.

[0088] In the embodiment of the present application, the query data encoding is input into the query time prediction model, the query prediction time is output, and the query result is determined based on the query prediction time and the preset query plan algorithm. It can be understood that the preset query plan algorithm can determine at least one query plan based on the query data, and then determine the optimal query plan based on the query prediction time and execute it, and then the query result can be determined.

[0089] The query time prediction model is obtained by training based on multiple query time prediction samples, and the query time prediction samples are determined based on a preset query plan algorithm and historical query data. Specifically, the preset query plan algorithm can be the above-mentioned query plan algorithm, and multiple executable query plans can be determined based on historical query data and the query plan algorithm. After executing multiple executable query plans, the actual execution time can be determined, and then multiple query time prediction samples are determined based on historical query data and the actual execution time, and the query time prediction model is obtained by training based on multiple query time prediction samples.

[0090] The reasoning query method containing a proxy model based on data coding provided by the embodiment of the present application encodes the query text and the query function separately, and determines the query data coding after splicing, which can at least reflect the query data distribution law in the query data coding, and can also reflect the correlation between the query functions if there are multiple query functions. The query data coding is then input into the query time prediction model, and the query prediction time is output, so that the prediction of the query time is more accurate. The query result is then determined based on the query prediction time and the preset query plan algorithm, which can more effectively determine the optimal data query plan, reduce the time of data query, and effectively improve the query efficiency.

[0091] The following describes the training method of the query time prediction model through the embodiments of the present application.

[0092] In some embodiments, before obtaining the query data, the method further includes:

[0093] S501: Acquire historical query data, and determine at least one data query plan based on the historical query data and a preset query plan algorithm.

[0094] S503: Execute at least one data query plan and determine the execution time corresponding to each data query plan.

[0095] S505: Determine a query time prediction sample based on historical query data, execution time, and a preset query plan algorithm.

[0096] In an embodiment of the present application, after obtaining historical query data and inputting the historical query data into a preset query plan algorithm, at least one data query plan can be determined. The preset query plan algorithm can refer to the above-mentioned query plan algorithm. By executing at least one data query plan, the actual execution time corresponding to each data query plan can be determined respectively. Based on the historical query data, the execution time and the preset query plan algorithm, the query time prediction sample can be further determined.

[0097] In some embodiments, the historical query data includes historical query text and historical query function data, such as Figure 3 As shown, the query time prediction samples determined based on historical query data, execution time, and a preset query plan algorithm include:

[0098] S601: Based on historical query data and a preset query plan algorithm, query plan parameters are determined, and the query plan parameters are matched with the data query plan.

[0099] S603: Determine a sample text encoding based on the historical query text, and determine a sample query parameter encoding based on the historical query function data and query plan parameters.

[0100] S605: Determine the sample query data code based on the sample text code and the sample query parameter code.

[0101] S607: Determine a query time prediction sample based on the sample query data encoding and execution time.

[0102] In the embodiment of the present application, the historical query data includes historical query text and historical query function data. The historical query function data includes a UDF containing a machine learning model. According to the above query plan algorithm, at least one data query plan can be generated based on the historical query data on different batches of historical query data. At least one data query plan on different batches of historical query data is executed to determine the execution results on different batches of historical query data. During the execution of the query plan algorithm, query plan parameters can be determined. Specifically, the query plan parameters can include α i ,s i , r i , c i Equal parameters, α i ,s i , r i , c i The meanings of the parameters can be found in the above descriptions of formula (1), formula (2), and formula (3), which will not be repeated here.

[0103] In some specific embodiments, the historical query data includes user-defined functions (UDFs) As well as historical custom query type parameters, historical custom query type parameters can include machine learning models and corresponding predicates. The data query plan includes multiple user-defined functions (UDFs) and their corresponding filters {σ1,…σ n} in the order of execution.

[0104] The sample text encoding is determined based on the historical query text, the sample query parameter encoding is determined based on the historical query function data and query plan parameters, and then the sample query data encoding is determined based on the sample text encoding and the sample query parameter encoding. Finally, the query time prediction sample can be determined based on the sample query data encoding and execution time.

[0105] In some embodiments, the historical query text includes a plurality of historical text data, and determining the sample text encoding based on the historical query text includes:

[0106] S701: Obtain a preset number of to-be-encoded texts from a plurality of historical text data, wherein the semantic order of the preset number of to-be-encoded texts is consistent with the semantic order of a plurality of historical text data in a historical query text.

[0107] S703: splicing a preset number of to-be-encoded texts based on semantic order to obtain a to-be-encoded text segment.

[0108] S705: Input the text segment to be encoded into a preset text embedding model to generate a sample embedding vector as the sample text encoding.

[0109] In an embodiment of the present application, the historical query text includes a plurality of historical text data, and a preset number of texts to be encoded are obtained from the plurality of historical text data, and the semantic order of the preset number of texts to be encoded is kept consistent with the semantic order in the historical query text. Specifically, if the order of text to be encoded 1 in the historical query text is less than the order of text to be encoded 2 in the historical query text, then the order of text to be encoded 1 in the set of texts to be encoded consisting of a preset number of texts to be encoded is also less than the order of text to be encoded 2. In some specific embodiments, a preset number of texts to be encoded can be obtained from a plurality of historical text data by random sampling.

[0110] A preset number of texts to be encoded are concatenated based on semantic order to obtain a text segment to be encoded. Specifically, a preset number of texts to be encoded are concatenated one by one to obtain a concatenated "text paragraph" p. The text segment to be encoded is input into a text embedding model to generate a sample embedding vector as a sample text encoding. Specifically, a text embedding model with a larger input sequence length can be selected, for example, a text embedding model with an input sequence length greater than 5000.

[0111] In some embodiments, the historical query function data includes a historical custom function and a historical custom query type parameter, and determining the sample query parameter code based on the historical query function data and the query plan parameter includes:

[0112] S801: Based on a one-hot encoding rule, historical custom functions, historical custom query type parameters, and query plan parameters are respectively encoded and concatenated based on a first preset concatenation order to obtain a sample query parameter encoding.

[0113] In the embodiment of the present application, based on the one-hot encoding rule, after encoding the historical custom function, the historical custom query type parameter, and the query plan parameter respectively, they are spliced ​​based on the first preset splicing order to obtain the sample query parameter encoding. Specifically, the first preset splicing order can be determined according to any order among the historical custom function, the historical custom query type parameter, and the query plan parameter, such as historical custom function-historical custom query type parameter-query plan parameter, or query plan parameter-historical custom query type parameter-historical custom function, etc.

[0114] In some embodiments, based on the sample text encoding and the sample query parameter encoding, determining the sample query data encoding includes:

[0115] S901: Based on a second preset concatenation order, concatenate the sample text code and the sample query parameter code to obtain a sample query data code.

[0116] In the embodiment of the present application, after determining the sample text code and the sample query parameter code, the sample query data code can be obtained by splicing them based on the second preset splicing order. Specifically, the second preset splicing order can include sample text code-sample query parameter code, or sample query parameter code-sample text code.

[0117] After determining the query time prediction sample, the query time prediction model can be trained using the query time prediction sample. In some embodiments, after determining the query time prediction sample based on the sample query data encoding and execution time, the method further includes:

[0118] S1001: construct a query time prediction model, in which training parameters are set.

[0119] S1003: Encode the sample query data and input it into the query time prediction model to generate a query time prediction result.

[0120] S1005: Based on the difference between the query time prediction result and the execution time, iteratively adjust the training parameters until the difference meets the preset requirements.

[0121] In the embodiment of the present application, a query time prediction model is constructed based on a machine learning method. The machine learning method may include a deep learning method, a reinforcement learning method, etc., and the generated model component may include a convolutional neural network model component, a recurrent neural network model component, etc., which is not limited in the present application. Specifically, the query time prediction model may include a convolutional neural network (CNN) for processing one-dimensional input data, which gradually extracts features through a series of convolutional layers, batch normalization layers, activation functions, and pooling layers, and finally regresses through a fully connected layer. After the query time prediction model is constructed, the sample query data encoding is input into the query time prediction model, and the corresponding query time prediction result is output. In the training process of the query time prediction model, the training parameters in the query time prediction model are continuously iterated and adjusted until the preset requirements are met. Meeting the preset requirements can be comparing the execution time with the query time prediction result, and when the difference is less than the preset threshold, it is considered that the preset requirements are met. It can also be the number of iterations of the preset training process, and when the number of iterations meets the preset value, it is considered that the preset requirements are met.

[0122] The following describes a specific embodiment of the present application to illustrate the training method of the query time prediction model.

[0123] Get the UDF filtering query containing the machine learning model, and use the above query plan algorithm to get the approximate optimal query plan on different batches of historical query data. Execute the approximate optimal query plan on different batches of historical query data, obtain the execution results on different batches of historical query data, and then determine the historical query data involved in the query, execution time and other information. The execution process of the algorithm is as follows:start_count,end_count,exec_start_count,exec_end_count,vcpu,available_cpus,available_memory,workflow_index,ml_operator_name_list,predicate_list,query_target_accuracy,pp1_index,pp1_train_cost,pp1_infer_cost,pp1_target_accuracy,pp1_final_accuracy,pp1_threshold,pp1_reduction,op1_name,op1_cost,op1_selectivity,pp2_index,pp2_tra in_cost,pp2_infer_cost,pp2_target_accuracy,pp2_final_accuracy,pp2_threshold,pp2_reduction,op2_name,op2_cost,op2_selectivity,pp3_index,pp3_train_cost,pp3_infer_cost,pp3 _target_accuracy,pp3_final_accuracy,pp3_threshold,pp3_reduction,op3_name,op3_cost,op3_selectivity,process_count,core_opt_time,c_batch_selectivity,c_batch_process_time.

[0124] 0,5500,0,1000,4,16,24.77,75,"['TextTCLSNerTweebankNLPHF','TextTCLSNerDslimBaseHF']","[""['B-PER','I-PER','B-LOC','B-MISC']"",""['I-ORG','B-ORG','B-L OC','B-PER','I-LOC','B-MISC']""]",0.9,1,9.192214012145996,6.98089599609375 e-05,0.9933774834437086,0.9921875,0.17535687060777116,0.03,TextTCLSNerTweeb ankNLPHF,0.01853668286408852,0.36474586744940524,27,4.815130949020386,5.71 4802802363529e-05,0.9035714285714286,0.8937728937728938,0.5568150197102085, 0.163727959697733,TextTCLSNerDslimBaseHF,0.03596064727753401,0.69661458333 33334,0,0,0,1,1,0,0,nothing,0,1,5500,757.1480515003204,0.221,91.9148380756.

[0125] Encode the historical query data and construct a sample data set consisting of multiple query time prediction samples. First, use the embedding model to process the "paragraphs" formed by splicing multiple texts to be encoded, and obtain the encoding of semantic information at the sample data set level as the sample text encoding. The processing data range can be 0-1000, and 100 data are randomly extracted and spliced ​​in semantic order. Specifically, if the current processing data range is 0-1000 texts, 100 data are randomly extracted, and a sample text encoding can be obtained by extracting once. Multiple operations on different queries and different batches of data can obtain multiple sample text encodings. The specific size of the sample data set can be set according to the actual situation. 1000 texts can be: {"id":0,"text":"@MarenticC@ESPNCFB you can't run on them.And whenJake Coker looks like Brady Bama beats the Browns"}

[0126] {"id":1,"text":"Happy new.Years Fam"}

[0127] {"id":2,"text":"#Hello2016"}

[0128] {"id":3,"text":"Happy New Year my ET friends! We're gonna be there soon.\ud83c\udf7e\ud83c\udf89"}

[0129] {"id":4,"text":"@KyleKulinski at least u tried"}

[0130] {"id":5,"text":"Happy New Years!"}

[0131] ....×1000.

[0132] The process of randomly extracting 100 pieces of data and concatenating them can be:

[0133] Happy New Years! thanks to 2015 for bringing some new people into my life. ....

[0135] Does ABC 7NYE program ever change?

[0136] ....×100.

[0137] Input the concatenated "text paragraph" into the text embedding model (such as nomic-embed-text-v1.5, the maximum input token number is 8192) to obtain the embedding vector:

[0138] -0.11916019,-0.01368711,0.10889089,0.4187666,-0.06526506,-0.02836056,0.02159151,0.13857684,....

[0139] Then, according to the one-hot encoding rule, the UDFs, predicates, and query plan parameters are encoded: 0.03191489, 0.01060596, 0.66339066, 271.26581407, 0.294, 0.27569488, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 1 ...

[0140] The sample text encoding is concatenated with the sample query parameter encoding to obtain a sample query data encoding. By repeating the above process, a complete sample data set is constructed, and the sample data set is used to train the deep learning model to obtain a query time prediction model.

[0141] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0142] Based on the same inventive concept, the embodiment of the present application also provides a data-encoded inference query device 1300 containing a proxy model for implementing the above-mentioned data-encoded inference query method containing a proxy model. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiments of one or more data-encoded inference query devices 1300 containing a proxy model provided below can refer to the limitations of the data-encoded inference query method containing a proxy model above, and will not be repeated here.

[0143] In one embodiment, Figure 4 As shown, a reasoning query device 1300 containing a proxy model based on data encoding is provided, comprising:

[0144] The query data acquisition module 1301 is used to acquire query data, where the query data includes query text and query function data;

[0145] An encoding module 1302, configured to determine a text encoding based on the query text, and to determine a query parameter encoding based on the query function data;

[0146] A query data code determination module 1303, configured to determine the query data code based on the text code and the query parameter code;

[0147] Query module 1304 is used to encode query data and input it into a query time prediction model, output the query prediction time, and determine the query result based on the query prediction time and a preset query plan algorithm, wherein the query time prediction model is trained based on multiple query time prediction samples, and the query time prediction samples are determined based on a preset query plan algorithm and historical query data.

[0148] In some embodiments, the reasoning query device 1300 containing a proxy model based on data encoding also includes a model training module 1305, which is used to obtain historical query data and determine at least one data query plan based on the historical query data and a preset query plan algorithm; execute at least one data query plan to determine the execution time corresponding to each data query plan; and determine a query time prediction sample based on the historical query data, the execution time and the preset query plan algorithm.

[0149] In some embodiments, historical query data includes historical query text and historical query function data. The model training module 1305 is also used to determine query plan parameters based on the historical query data and a preset query plan algorithm, and match the query plan parameters with the data query plan; determine a sample text encoding based on the historical query text, and determine a sample query parameter encoding based on the historical query function data and the query plan parameters; determine a sample query data encoding based on the sample text encoding and the sample query parameter encoding; and determine a query time prediction sample based on the sample query data encoding and the execution time.

[0150] In some embodiments, the historical query text includes multiple historical text data, and the model training module 1305 is also used to obtain a preset number of texts to be encoded from the multiple historical text data, and the semantic order of the preset number of texts to be encoded is consistent with the semantic order of multiple historical text data in the historical query text; the preset number of texts to be encoded are spliced ​​based on the semantic order to obtain a text segment to be encoded; the text segment to be encoded is input into a preset text embedding model to generate a sample embedding vector as a sample text encoding.

[0151] In some embodiments, historical query function data includes historical custom functions and historical custom query type parameters. The model training module 1305 is also used to encode the historical custom functions, historical custom query type parameters and query plan parameters respectively based on the one-hot encoding rule and splice them based on the first preset splicing order to obtain sample query parameter encoding.

[0152] In some embodiments, the model training module 1305 is further used to concatenate the sample text code and the sample query parameter code based on a second preset concatenation order to obtain the sample query data code.

[0153] In some embodiments, the model training module 1305 is also used to construct a query time prediction model, in which training parameters are set; the sample query data encoding is input into the query time prediction model to generate a query time prediction result; based on the difference between the query time prediction result and the execution time, the training parameters are iteratively adjusted until the difference meets the preset requirements.

[0154] In some embodiments, the query function data includes a custom function and a custom query type parameter. The encoding module 1302 is also used to input the query text into a preset text embedding model to generate a query embedding vector as a text encoding; based on the one-hot encoding rule, the custom function and the custom query type parameters are encoded separately, and then spliced ​​based on a third preset splicing order to obtain the query parameter encoding.

[0155] In some embodiments, the query data code determination module 1303 is further configured to concatenate the text code and the query parameter code based on a fourth preset concatenation order to obtain the query data code.

[0156] Each module in the above-mentioned reasoning query device 1300 containing a proxy model based on data encoding can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0157] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a reasoning query method containing an agent model based on data encoding is implemented.

[0158] Those skilled in the art will understand that Figure 5The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0159] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the reasoning query method containing a proxy model based on data encoding in any of the above embodiments are implemented.

[0160] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the reasoning query method containing a proxy model based on data encoding in any of the above embodiments.

[0161] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0162] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0163] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0164] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A reasoning query method containing a proxy model based on data encoding, characterized in that: The method comprises: Acquire query data, wherein the query data includes query text and query function data; determining a text encoding based on the query text, and determining a query parameter encoding based on the query function data; Determine the query data code based on the text code and the query parameter code; The query data encoding is input into a query time prediction model, the query prediction time is output, and the query result is determined based on the query prediction time and a preset query plan algorithm, wherein the query time prediction model is trained based on multiple query time prediction samples, and the query time prediction samples are determined based on the preset query plan algorithm and historical query data.

2. The method according to claim 1, characterized in that Before obtaining the query data, the method further includes: Acquire the historical query data, and determine at least one data query plan based on the historical query data and the preset query plan algorithm; Execute the at least one data query plan and determine the execution time corresponding to each data query plan; The query time prediction sample is determined based on the historical query data, the execution time, and the preset query plan algorithm.

3. The method according to claim 2, characterized in that The historical query data includes historical query text and historical query function data, and determining the query time prediction sample based on the historical query data, the execution time and the preset query plan algorithm includes: Determine query plan parameters based on the historical query data and the preset query plan algorithm, wherein the query plan parameters match the data query plan; Determine a sample text code based on the historical query text, and determine a sample query parameter code based on the historical query function data and the query plan parameters; Determine a sample query data code based on the sample text code and the sample query parameter code; The query time prediction sample is determined based on the sample query data encoding and the execution time.

4. The method according to claim 3, characterized in that The historical query text includes a plurality of historical text data, and determining the sample text encoding based on the historical query text includes: Obtaining a preset number of to-be-encoded texts from the plurality of historical text data, wherein the semantic order of the preset number of to-be-encoded texts is consistent with the semantic order of the plurality of historical text data in the historical query text; splicing the preset number of to-be-encoded texts based on the semantic order to obtain a to-be-encoded text segment; The text segment to be encoded is input into a preset text embedding model to generate a sample embedding vector as the sample text encoding.

5. The method according to claim 3, characterized in that: The historical query function data includes a historical custom function and a historical custom query type parameter, and the sample query parameter code is determined based on the historical query function data and the query plan parameter, including: Based on the one-hot encoding rule, the historical custom function, the historical custom query type parameter, and the query plan parameter are respectively encoded and concatenated based on a first preset concatenation order to obtain the sample query parameter encoding.

6. The method according to claim 3, characterized in that The determining of the sample query data code based on the sample text code and the sample query parameter code comprises: Based on a second preset concatenation order, the sample text code and the sample query parameter code are concatenated to obtain the sample query data code.

7. The method according to claim 3, characterized in that After determining the query time prediction sample based on the sample query data encoding and the execution time, the method further includes: Constructing a query time prediction model, wherein the query time prediction model is provided with training parameters; Inputting the sample query data encoding into the query time prediction model to generate a query time prediction result; Based on the difference between the query time prediction result and the execution time, the training parameters are iteratively adjusted until the difference meets a preset requirement.

8. The method according to claim 1, characterized in that The query function data includes a custom function and a custom query type parameter, and the determining of the text encoding based on the query text and the determining of the query parameter encoding based on the query function data include: Inputting the query text into a preset text embedding model to generate a query embedding vector as the text encoding; Based on the one-hot encoding rule, the user-defined function and the user-defined query type parameters are encoded respectively, and then spliced ​​based on a third preset splicing order to obtain the query parameter encoding.

9. The method according to claim 8, characterized in that The obtaining of the query data code based on the text code and the query parameter code comprises: Based on a fourth preset concatenation order, the text code and the query parameter code are concatenated to obtain the query data code.

10. A reasoning query device containing a proxy model based on data encoding, characterized in that: The device comprises: A query data acquisition module, used to acquire query data, wherein the query data includes query text and query function data; An encoding module, used to determine a text encoding based on the query text, and to determine a query parameter encoding based on the query function data; A query data code determination module, used to determine the query data code based on the text code and the query parameter code; A query module is used to encode the query data and input it into a query time prediction model, output the query prediction time, and determine the query result based on the query prediction time and a preset query plan algorithm, wherein the query time prediction model is trained based on multiple query time prediction samples, and the query time prediction samples are determined based on the preset query plan algorithm and historical query data.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images