Large language model-based query statement generation method, apparatus, and device, and medium

Abstract

The present application provides a large language model-based query statement generation method, apparatus, and device, and a medium. The large language model comprises a plurality of encoders. The method comprises: acquiring a query text; performing vectorization processing on the query text to obtain a target vector representation corresponding to the query text; inputting the target vector representation into the plurality of encoders, to perform encoding processing on the target vector representation by means of weight matrices of the encoders, so as to obtain context vectors corresponding to the encoders, wherein different encoders have different weight matrices; and generating a target query statement on the basis of the context vectors corresponding to the encoders. A model is allowed to focus on different information features in different attention heads, independently capture different aspects of a query text, and process information at multiple abstraction levels, thereby better capturing complex and abstract semantic relationships, effectively distinguishing the importance of information when processing complex and fuzzy questions, improving the accuracy of query statement generation.

Description

Method and device for generating query statement based on large language model, equipment and medium

[0001] The present application claims priority to the Chinese patent application No. 202411008424.2, filed on July 25, 2024, and entitled "Method and device for generating query statement based on large language model, equipment and medium", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of big data, in particular to a method and device for generating query statement based on large language model, equipment and medium. BACKGROUND

[0003] With the rapid development of information technology, databases have become an important part of our daily life and work. How to make ordinary users more convenient to access and use the information in the database has become an important research topic. Text-to-structured query language (text-to-SQL) converts the user's natural language query into a corresponding structured query language (SQL) query statement through a large language model (LLM), so that users can directly interact with the database through natural language.

[0004] In related technologies, text-to-SQL mostly uses a general Transformer large model for processing. When processing the text-to-SQL task, the general Transformer large model encodes the user's input natural language query text and the database into embedding vectors, and performs context fusion and self-attention mechanism processing through a traditional Transformer structure to capture the relationship between the text and the database structure. Finally, a decoder is used to generate a SQL query statement that meets the syntax rules, and a loss function is used to continuously optimize the model weights during the training process to improve the generation accuracy.

[0005] However, the accuracy of the query statement generation of the prior art is not high. SUMMARY

[0006] The present application provides a method and device for generating query statement based on large language model, equipment and medium, thereby solving the problem of low accuracy of query statement generation in the prior art.

[0007] In a first aspect, the present application provides a method for generating query statement based on large language model, the large language model comprising a plurality of encoders, the method comprising:

[0008] obtaining a query text;

[0009] vectorize the query text to obtain a target vector representation corresponding to the query text;

[0010] input the target vector representation into the plurality of encoders to encode the target vector representation through a weight matrix of each encoder to obtain a context vector corresponding to each encoder, wherein the weight matrix of different encoders is different;

[0011] generate a target query sentence according to the context vector corresponding to each encoder.

[0012] The application provides a query sentence generation method based on a large language model. The method generates a query sentence by using a large language model based on a multiple attention mechanism. The large language model is improved on the basis of a general Transformer large model and is provided with a plurality of encoders. For a target vector representation corresponding to a query text, the target vector representation is encoded by the plurality of encoders of the large language model based on a weight matrix in each encoder. A context vector is output by each encoder. Then, a target query sentence is generated based on the context vectors output by different encoders. Since the weight matrix in each encoder is different, each encoder can pay attention to different semantic and structural features of the query text when extracting features. The model can generate specific templates in different subspaces, allow the model to focus on different information features in different attention heads, independently capture different aspects of the query text, process information at multiple abstraction levels, better capture complex and abstract semantic relationships, and better distinguish the importance of information when processing complex and ambiguous questions. Therefore, the accuracy of query sentence generation is improved.

[0013] Optionally, the encoder comprises a first self-attention layer and a context vector layer. The inputting of the target vector representation into the plurality of encoders to encode the target vector representation through a weight matrix of each encoder to obtain a context vector corresponding to each encoder comprises:

[0014] inputting the target vector representation into the first self-attention layer of each encoder to calculate the target vector representation through a weight matrix of the first self-attention layer of each encoder to obtain a plurality of output vectors of each encoder; and inputting the plurality of output vectors of each encoder into a corresponding context vector layer to weight the plurality of output vectors through the context vector layer to obtain a context vector corresponding to each encoder.

[0015] Here, the encoder of the present application includes a self-attention layer for assigning different attention weights to the target vector representation to extract semantic information selectively, and a context vector layer for weighted processing to obtain a context vector, the self-attention layer is used to focus on different semantic features, and flexibly and accurately identify multiple key information in the query text, and the context vector layer can output a context vector to help the large model focus on the relevant part of the encoder output, further improving the accuracy of the query statement generation.

[0016] Optionally, before the target vector representation is input into the first self-attention layer of each of the plurality of encoders to calculate the target vector representation through the weight matrix of the first self-attention layer of each of the plurality of encoders to obtain a plurality of output vectors of each of the plurality of encoders, the method further comprises:

[0017] For each encoder, a training sample corresponding to the encoder is obtained, wherein the training sample includes an input data sample and a label sample; the training sample is input into the first self-attention layer for forward propagation to obtain an actual output corresponding to the input data sample; a loss gradient is obtained according to a preset loss function, the actual output and the label sample; the weight matrix of the first self-attention layer is updated according to the loss gradient using a back propagation algorithm until the weight matrix meets an error requirement.

[0018] Wherein, the traditional Transformer model uses a self-attention mechanism to capture the relationship within the sequence, and performs a weighted average on all key-value pairs without a clear way to distinguish which is key information and which is interference information. For ambiguous query intentions, the model may not be able to accurately identify the key information because it relies on a single attention mechanism to process all types of semantic information. The present application can solve the above problems by introducing multiple attention encoders based on a multiple attention mechanism, each of which learns different representations of the input sequence. In order to implement the multiple attention mechanism, the present application pre-acquires a corresponding training sample for each encoder, trains the encoder using a preset loss function, and then obtains the weight matrix of each encoder. Different weight matrices trained by different training samples can focus on different information features during query statement generation, further improving the accuracy of query statement generation.

[0019] Optionally, the weight matrix includes self-attention weights, feedforward network weights, layer normalization weights, and position encoding weights.

[0020] Wherein, each encoder of the present application has an independent weight matrix, including self-attention weights, feedforward network weights, and layer normalization and position encoding weights, and each encoder can more accurately focus on different information.

[0021] Optionally, the generating the target query statement according to the target context vector comprises: performing concatenation processing on the context vectors corresponding to the encoders to obtain a target context vector; and generating the target query statement according to the target context vector.

[0022] In the present application, the context vectors obtained by different encoders are concatenated to obtain a target context vector including all context vectors, and the target query statement is generated according to the target context vector. The target context vector includes information obtained by different attention head angle analysis. Even if there is interference information, some attention heads can recognize and focus on key information, allowing the model to analyze and process the query text from different angles. When processing ambiguous queries, different attention heads may capture different clues to help the model better understand the query intent, improving the accuracy of query statement generation.

[0023] Optionally, the generating the target query statement according to the target context vector comprises: performing concatenation processing on the context vectors corresponding to the encoders to obtain a target context vector; and generating the target query statement according to the target context vector.

[0024] In the present application, the context vectors obtained by different encoders are concatenated to obtain a target context vector including all context vectors, and the target query statement is generated according to the target context vector. The target context vector includes information obtained by different attention head angle analysis. Even if there is interference information, some attention heads can recognize and focus on key information, allowing the model to analyze and process the query text from different angles. When processing ambiguous queries, different attention heads may capture different clues to help the model better understand the query intent, improving the accuracy of query statement generation.

[0025] Optionally, the large language model further comprises a query intent recognizer, and the query intent recognizer is located in the decoder of the large language model.

[0026] The decoding processing of the target context vector to obtain the structured query language template comprises: inputting the target context vector into the query intent recognizer to perform query intent recognition processing through the query intent recognizer to obtain at least one key information; and determining a structured query language mark of the structured query language template according to the at least one key information.

[0027] Here, the present application sets a query intent recognizer, which is a module specially used for identifying user query intent. It is crucial for generating accurate SQL templates. In the text-to-SQL task, the natural language query of the user may contain multiple intents. The role of the query intent recognizer is to determine the query intent of the user, i.e., the key information, so as to generate a correct SQL statement. In the decoder, the query intent recognizer can help the model select the correct structure and keywords when generating the SQL template, improving the accuracy of query statement generation.

[0028] Optionally, the target context vector is input into the query intent recognizer to perform query intent recognition processing by the query intent recognizer to obtain at least one key information, comprising: inputting the target context vector into the query intent recognizer; in the query intent recognizer, according to a preset weight value, a preset bias value and a preset activation function in a preset intent recognition function, the target context vector is calculated to obtain at least one key information.

[0029] Optionally, the large language model further comprises a decoder module and a template guided attention module, and the decoder module and the template guided attention module are located in the decoder of the large language model; the decoding processing of the target context vector to obtain the structured query language template comprises: inputting the target context vector into the decoder module for decoding processing, and obtaining a plurality of decoder outputs of the decoder module according to time steps; inputting the plurality of decoder outputs into the template guided attention module; in the template guided attention module, a predefined structured query language template is obtained; in the template guided attention module, according to the predefined structured query language template and the plurality of decoder outputs, the structure of the structured query language template is determined.

[0030] In this application, a template guided attention module is also provided, which is a module specially designed to guide the Transformer model to follow a specific structure when generating SQL templates. This module can help the model generate structured SQL statements during the decoding process, especially when dealing with ambiguous queries. It can provide guidance when the decoder generates SQL statements, allowing the model to dynamically adjust the generated SQL template according to the input natural language query. This can effectively handle ambiguous queries and further improve the accuracy of query statement generation.

[0031] Optionally, in the template guided attention module, according to the predefined structured query language template and the plurality of decoder outputs, the structure of the structured query language template is determined, comprising: in the template guided attention module, performing linear transformation processing on each of the decoder outputs to obtain a plurality of linear transformation outputs; in the template guided attention module, performing linear transformation processing on the predefined structured query language template to obtain a linear transformation template; in the template guided attention module, according to a preset calculation function, each of the linear transformation outputs and the linear transformation template is calculated to obtain a plurality of attention outputs; in the template guided attention module, the plurality of attention outputs are spliced to obtain the structure of the structured query language template.

[0032] Based on the above method, the current output of the decoder can be aligned with the corresponding part in the predefined SQL template, focusing on guiding the decoder to generate the correct SQL structure.

[0033] Optionally, the decoder module comprises a second self-attention layer, a decoding attention layer and a forward propagation module.

[0034] Optionally, the filling processing of the structured query language template to obtain the target query statement comprises: identifying a placeholder in the structured query language template; obtaining a preset candidate set, wherein the preset candidate set comprises a plurality of replacement items; and filling processing the placeholder in the structured query language template according to the preset candidate set to obtain the target query statement.

[0035] Optionally, the filling processing of the structured query language template to obtain the target query statement according to the preset candidate set comprises: using a beam search algorithm to screen the replacement items in the preset candidate set multiple times, and filling the screened replacement items to the placeholder in the structured query language template until all the placeholders are replaced.

[0036] In this application, the beam search algorithm is used to fill the generated structured SQL template to generate a specific SQL query. The beam search algorithm can maintain a certain "width" in the search space by retaining the most likely candidate results at each step, while limiting the "depth" of the search, to balance the breadth and depth of the search, reduce the amount of calculation, and improve the search efficiency. By combining the template generation based on the multiple attention mechanism and the template filling based on the beam search, this application can generate diversified SQL queries while maintaining the rationality of the query structure, improving the relevance and accuracy of the generated queries.

[0037] Optionally, the obtaining of the preset candidate set comprises: querying a preset database to obtain the preset candidate set; and / or obtaining historical query data of a user corresponding to the query text, and generating the preset candidate set based on the historical query data.

[0038] Here, the preset candidate set for filling the SQL template can be established based on the preset database or the historical query data of the user corresponding to the query text, which not only enables accurate SQL template filling through the massive information in the preset database, but also enables customized SQL statement generation based on the user, further improving the flexibility and accuracy of SQL statement generation and improving the user experience.

[0039] Optionally, the vectorizing the query text to obtain the target vector representation corresponding to the query text comprises: inputting the query text into a pre-trained language model to obtain the target vector representation corresponding to the query text through an output of the pre-trained language model.

[0040] In this application, the target vector representation corresponding to the query text can be extracted based on the pre-trained language model, and the target vector representation of the query text can be efficiently and accurately extracted, thereby providing an accurate and effective basis guarantee for the generation of the query sentence.

[0041] Optionally, the large language model further comprises an input embedding layer and a first linear function layer; and the vectorizing the query text to obtain the target vector representation corresponding to the query text comprises: inputting the query text into the input embedding layer to output an embedding vector corresponding to the query text through the input embedding layer; and inputting the embedding vector into the first linear function layer to output the target vector representation corresponding to the query text through the first linear function layer.

[0042] Here, the input query text is converted into a vector representation in a high-dimensional space through the input embedding layer, and is then transformed into a desired dimension through the linear function layer in the encoder, thereby facilitating subsequent processing.

[0043] In a second aspect, the application provides a query sentence generation device based on a large language model, wherein the large language model comprises a plurality of encoders, and the device comprises:

[0044] The acquisition module is configured to acquire a query text.

[0045] The first processing module is configured to vectorize the query text to obtain a target vector representation corresponding to the query text.

[0046] The second processing module is configured to input the target vector representation into the plurality of encoders to encode the target vector representation through a weight matrix of each encoder to obtain a context vector corresponding to each encoder, wherein the weight matrix of different encoders is different.

[0047] The generation module is configured to generate a target query sentence according to the context vector corresponding to each encoder.

[0048] In a third aspect, the application provides a query sentence generation device based on a large language model, comprising at least one processor and a memory.

[0049] The memory stores computer execution instructions.

[0050] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the large language model-based query sentence generation method as described in the first aspect and various possible designs of the first aspect.

[0051] In a fourth aspect, the present application provides a computer-readable storage medium, which stores computer-executable instructions, and when a processor executes the computer-executable instructions, the large language model-based query sentence generation method as described in the first aspect and various possible designs of the first aspect is implemented.

[0052] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and when a processor executes the computer program, the large language model-based query sentence generation method as described in the first aspect and various possible designs of the first aspect is implemented.

[0053] The large language model-based query sentence generation method, device, equipment and medium provided by the present application are provided, wherein the method generates a query sentence by using a large language model. The large language model is improved on the basis of a general Transformer large model, and multiple encoders are set. For a corresponding target vector representation of a query text, the target vector representation is encoded and processed based on a weight matrix in each encoder through the multiple encoders of the large language model, and a context vector is output after each encoder. Then, the target query sentence is generated based on the context vectors output by different encoders. Since the weight matrices in each encoder are different, each encoder can pay attention to different semantic and structural features of the query text when extracting features, so that the model can generate specific templates in different subspaces, allow the model to focus on different information features in different attention heads, independently capture different aspects of the query text, process information at multiple abstraction levels, and thus better capture complex and abstract semantic relationships. When processing complex and ambiguous questions, the importance of information can be well distinguished, and the accuracy of query sentence generation is improved. BRIEF DESCRIPTION OF DRAWINGS

[0054] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0055] FIG. 1 is a structural schematic diagram of a large language model-based query sentence generation system according to an embodiment of the present application;

[0056] FIG. 2 is a structural schematic diagram of a large language model according to an embodiment of the present application;

[0057] FIG. 3 is a flowchart of a large language model-based query sentence generation method according to an embodiment of the present application;

[0058] FIG. 4 is a flow diagram of another method for generating a query statement based on a large language model according to an embodiment of the present application;

[0059] FIG. 5 is a flow diagram of another method for generating a query statement based on a large language model according to an embodiment of the present application;

[0060] FIG. 6 is a flow diagram of another method for generating a query statement based on a large language model according to an embodiment of the present application;

[0061] FIG. 7 is a structural diagram of another large language model according to an embodiment of the present application;

[0062] FIG. 8 is a structural diagram of a device for generating a query statement based on a large language model according to an embodiment of the present application;

[0063] FIG. 9 is a structural diagram of a device for generating a query statement based on a large language model according to an embodiment of the present application.

[0064] The specific embodiments of the present application have been shown through the above-described drawings, and will be described in more detail hereinafter. These drawings and written descriptions are not intended to limit the scope of the present application concept in any way, but to illustrate the present application concept to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0065] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is only one of the exemplary embodiments in which identical or similar components are denoted by the same reference numerals throughout the several views. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application.

[0066] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards, and provide corresponding operation portal for user to choose authorization or refusal.

[0067] In recent years, with the rapid development of large language models (LLMs), new solutions have been provided for text-to-SQL tasks. Large language models are natural language processing models trained using large-scale corpora through machine learning. Large language models can understand complex semantics and contextual information in natural language queries, which enables them to convert into corresponding SQL queries more accurately. General Transformer large models are widely used large language models. However, current text-to-SQL techniques based on large language models still face the following challenges: not supporting fuzzy queries: existing text-to-SQL user questions may not be clear or specific enough, at this time, the intention recognition of fuzzy questions is particularly important. By accurately understanding the user's query intention, the model can generate SQL queries that better meet the user's needs. Methods often have difficulty processing queries containing fuzzy matches, such as partial matches of keywords. In practical applications, users may ask very complex queries containing multiple conditions and subqueries. In such cases, the ability to identify intentions and filter irrelevant information is particularly important. Cannot exclude irrelevant text interference: In practical applications, user queries may contain a large amount of irrelevant information that will affect the accuracy of the query results. Users may include some words or details unrelated to database queries when asking questions. If the model cannot effectively identify and filter this information, the generated SQL statements may contain incorrect conditions or structures, resulting in incorrect results. In summary, the existing technology has the technical problem of low accuracy of query statement generation.

[0068] To solve the above problems, the embodiments of the present application provide a query statement generation method, device, equipment and medium based on a large language model. The method generates a query statement based on a large language model based on a multiple attention mechanism. The large language model is improved based on a general Transformer large model and sets multiple encoders. For the corresponding target vector representation of the query text, the target vector representation is encoded and processed based on the weight matrix in each encoder through the multiple encoders of the large language model. The context vector is output after each encoder, and the target query statement is generated based on the context vectors output by different encoders.

[0069] In one possible implementation, the general Transformer large model needs to be specially modified to better handle the text to SQL task. Although the general model has strong language understanding ability, it is not optimized for the specific semantics and structured output of SQL queries. The necessity of modification lies in the need for the model to accurately capture the intent of the user query, understand the complex associations of the database schema, and ensure the correctness of the generated SQL statements in terms of syntax and logic. The text to SQL task puts higher requirements on the decoding efficiency and long sequence processing capability of the model, therefore, through the customization of attention mechanism, adjustment of input and output layers, and training strategies tailored for SQL features, the performance of the model on the text to SQL task can be significantly improved, achieving more efficient and accurate query statement generation. The general Transformer model uses a self-attention mechanism to focus on all input sequence positions simultaneously, but this mechanism may not be able to well distinguish the importance of information when dealing with complex and ambiguous questions. The multiple attention mechanism proposed in the embodiments of the present application allows the model to focus on different information features in different attention heads. (1) Some attention heads can be dedicated to identifying and focusing on keywords, while other attention heads can be used to understand complex relationships or ambiguous expressions in the context. This fine-grained focus helps the model identify and prioritize key information while filtering out irrelevant content. (2) The multiple attention mechanism achieves multiple attention modes through different attention heads, wide attention (widely capturing context) and narrow attention (focusing on key information). This diversity helps the model better understand complex and ambiguous questions, as it can analyze the input text from multiple perspectives. Certain attention heads can be trained to specifically identify and interpret ambiguous expressions, such as metaphors, omissions, or ambiguous sentences. (3) The multiple attention mechanism can focus on different semantic or structural features through different attention heads. Even with interfering information, certain attention heads can identify and focus on key information. For example, one attention head may focus on entity recognition, while another focuses on semantic role labeling. The multiple attention mechanism allows the model to examine the input sequence from different perspectives. When dealing with ambiguous queries, different attention heads may capture different clues, helping the model better understand the query intent.

[0070] Optionally, FIG. 1 is a system architecture diagram of a query statement generation system based on a large language model according to an embodiment of the present application. In FIG. 1, the above architecture includes at least one of a data acquisition device 101, a processing device 102, and a display device 103.

[0071] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the architecture of the query sentence generation system based on the large language model. In other feasible embodiments of the present application, the above-mentioned architecture can include more or fewer components than the illustration, or combine certain components, or split certain components, or different arrangement of components, which can be determined according to the actual application scenario, and is not limited herein. The components shown in FIG. 1 can be realized by hardware, software, or a combination of software and hardware.

[0072] In the process of implementation, the data acquisition device 101 can include an input / output interface and can also include a communication interface. The data acquisition device 101 can be connected to the processing device through the input / output interface or the communication interface.

[0073] The processing device 102 can encode the target vector representation of the query text through a plurality of encoders of the large language model based on the weight matrix in each encoder respectively, output context vectors through each encoder, and generate a target query sentence based on the context vectors output by different encoders.

[0074] The display device 103 can also be a touch display screen or a screen of a terminal device, which is used to receive user instructions while displaying the above-mentioned content to realize interaction with the user.

[0075] It should be understood that the above-mentioned processing device can be realized by a processor reading instructions in a memory and executing the instructions, or can be realized by a chip circuit.

[0076] In addition, the network architecture and business scenarios described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. It can be known by those skilled in the art that, with the evolution of network architecture and the appearance of new business scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0077] The technical solutions of the present application will be described in detail below in combination with specific embodiments:

[0078] Optionally, FIG. 2 is a structure diagram of a large language model provided by an embodiment of the present application. As shown in FIG. 2, the large language model provided by the embodiment of the present application includes a first encoder 21, a second encoder 22 and a third encoder 23. Each encoder includes a weight matrix different from other encoders.

[0079] It can be understood that the large language model can also include other structures, which are not specifically shown in FIG. 1.

[0080] It can be understood that in the large language model, the number of encoders can be determined according to actual conditions, and the number of encoders is greater than 1, so as to realize a multiple attention mechanism through different encoding matrices in multiple encoders.

[0081] Optionally, based on the large language model in the embodiment of the present application, the target vector representation can be encoded and processed based on the weight matrix in each encoder through multiple encoders of the large language model, and the context vector is output after each encoder, and the target query statement is generated based on the context vectors output by different encoders.

[0082] Optionally, FIG. 3 is a flowchart of a query statement generation method based on a large language model according to an embodiment of the present application. The execution subject of the embodiment of the present application can be the query statement generation system based on the large language model in FIG. 1, and the specific execution subject can be determined according to the actual application scene. As shown in FIG. 3, the method comprises the following steps:

[0083] S301: Obtain a query text.

[0084] The query text herein can be a query question input by a user, for example: list all the user names and email addresses of users who have purchased product X.

[0085] S302: Vectorize the query text to obtain a target vector representation corresponding to the query text.

[0086] Optionally, the vectorization of the query text to obtain the target vector representation corresponding to the query text comprises: inputting the query text into a pre-trained language model to obtain the target vector representation corresponding to the query text through the output of the pre-trained language model.

[0087] Optionally, the pre-trained language model is a Bidirectional Encoder Representations from Transformers (BERT) model. The problem is encoded using a pre-trained natural language processing model (such as BERT) to obtain a vector representation that captures the semantic information of the problem. This vector representation will serve as the basis for the multiple attention mechanism.

[0088] In the embodiment of the present application, the target vector representation corresponding to the query text can be extracted based on the pre-trained language model, and the target vector representation of the query text can be efficiently and accurately extracted, thereby providing an accurate and effective basis for the generation of the query statement.

[0089] Optionally, the large language model further comprises an input embedding layer and a first linear function layer; the vectorization processing of the query text to obtain the target vector representation corresponding to the query text comprises: inputting the query text into the input embedding layer to output an embedding vector corresponding to the query text through the input embedding layer; and inputting the embedding vector into the first linear function layer to output the target vector representation corresponding to the query text through the first linear function layer.

[0090] The input embedding layer and the linear function layer are located before the encoder.

[0091] Input embedding layer: Since the machine learning model cannot directly understand the text data, the task of the input embedding layer is to convert the input words or characters into vector representations in a high-dimensional space, i.e., embeddings. Each word or character is assigned a unique vector that captures its semantic and syntactic information. Positional encoding is usually added to these embedding vectors to preserve the position information of the words in the original sequence, because the embeddings themselves do not contain position information.

[0092] Linear function layer: Linear functions in the model mainly exist before each sub-layer, which are used to transform the input into the desired dimension. These linear transformations are usually implemented by multiplying the input vector by a weight matrix and adding a bias vector. They are used in both encoder and decoder layers. Forward propagation refers to the process of input data passing through the entire model. Inside each layer, data will pass through multiple sub-layers, including self-attention layers and feed-forward neural networks.

[0093] Here, the embodiments of the present application convert the input query text into vector representations in a high-dimensional space through the input embedding layer, and then transform them into the desired dimension through the linear function layer in the encoder, which facilitates subsequent processing.

[0094] S303: input the target vector representation into multiple encoders to encode the target vector representation through the weight matrix of each encoder to obtain a context vector corresponding to each encoder.

[0095] The weight matrix of different encoders is different.

[0096] Among them, the multiple attention mechanism adopted by the embodiments of the present application mainly has the following principles:

[0097] The original Transformer model uses a self-attention mechanism to capture the relationship within the sequence. Self-attention is calculated by the similarity of Query and Key. For example, a dot product multiple attention mechanism can be used, whose formula is (1): Attention(Q,K,V)=softmax(qK TFormula (1)

[0098] In formula (1), Attention(Q,K,V) represents the attention solution result, Q is the query, K is the key, V is the value, and K is the value. T This refers to the key dimension. In this example, the query could be the current state of the decoder, and the keys and values ​​could be vector representations of the question encoding. The original Transformer's attention mechanism may not be robust enough when handling queries containing a lot of distracting information. This is because self-attention performs a weighted average of all key-value pairs, without a clear way to distinguish which information is key and which is distracting. For ambiguous query intents, the model may not be able to accurately identify key information because it relies on a single attention mechanism to handle all types of semantic information. Understandably, in any formula in this application, the specific meaning of the parameters and their representations only represents the interpretation in that formula and is not generalized to other formulas. For the specific meaning of the parameters in each formula, please refer to the explanation of each formula.

[0099] The multi-attention mechanism solves the above problem by introducing multiple attention encoders, each of which learns different representations of the input sequence, as shown in the following formula (2): MultiAttention(Q,K,V)=Concat(Encoder1,...,Encoder h )W formula (2)

[0100] In formula (2), MultiAttention(Q,K,V) represents the attention solution result, h is the number of attention encoders, Q is the query, K is the key, V is the value, and W is the weight matrix of each encoder. Each attention encoder has an independent weight matrix. Different weight matrices allow each attention encoder to learn in different subspaces, thereby improving the model's ability to identify key information.

[0101] S304: Generate the target query statement based on the context vector corresponding to each encoder.

[0102] The application provides a query sentence generation method based on a large language model. The method generates a query sentence by using a large language model based on a multiple attention mechanism. The large language model is improved on the basis of a general Transformer large model, and multiple encoders are arranged. The corresponding target vector representation of the query text is encoded and processed by the multiple encoders of the large language model based on the weight matrix in each encoder, and the context vector is output after each encoder. The target query sentence is generated based on the context vectors output by different encoders. Since the weight matrix in each encoder is different, each encoder can pay attention to different semantic and structural features of the query text when extracting features, so that the model can generate specific templates in different subspaces, allow the model to focus on different information features in different attention heads, independently capture different aspects of the query text, process information at multiple abstraction levels, better capture complex and abstract semantic relationships, and better distinguish the importance of information when processing complex and ambiguous questions, thereby improving the accuracy of query sentence generation.

[0103] Optionally, the encoder comprises a first self-attention layer and a context vector layer. Correspondingly, FIG. 4 is a flowchart of another query sentence generation method based on a large language model provided by an embodiment of the application, as shown in FIG. 4, the method comprises:

[0104] S401: obtaining a query text.

[0105] S402: performing vectorization processing on the query text to obtain a target vector representation corresponding to the query text.

[0106] The implementation manners of steps S401 and S402 are the same as those of steps S301 and S302, and will not be repeated here.

[0107] S403: inputting the target vector representation into the first self-attention layer of the multiple encoders to calculate the target vector representation by the weight matrix of the first self-attention layer of each encoder, and obtaining multiple output vectors of each encoder.

[0108] S404: inputting the multiple output vectors of each encoder into the corresponding context vector layer to perform weighted processing on the multiple output vectors by each context vector layer, and obtaining the context vector corresponding to each encoder.

[0109] where the weight matrices of different encoders are different. Self-attention layers allow the model to assign different weights to each word in the input sequence, based on the relationship of the word with all other words. Self-attention works by computing three vectors: Query, Key, and Value. Each input word generates these three vectors, then the dot product between all queries and keys is computed to get the attention weights, and finally these weights are used to weight the value vectors to generate the output vector. Scaled dot-product attention is a technique to prevent the gradient from vanishing or exploding due to large inner products.

[0110] Context vector layer: On the decoder side, each decoder contains a context vector (also known as the output of the "encoder-decoder attention"), which represents a weighted combination of the encoder output sequence. This context vector is used to generate the output of the next decoder. The context vector is a special kind of self-attention mechanism in the decoder, which helps the decoder focus on the relevant parts of the encoder output.

[0111] Optionally, before inputting the target vector representation into the first self-attention layer of each encoder to calculate the target vector representation through the weight matrix of the first self-attention layer of each encoder to obtain a plurality of output vectors of each encoder, further comprising:

[0112] For each encoder, obtain the training sample corresponding to the encoder, wherein the training sample includes an input data sample and a label sample; input the training sample into the first self-attention layer for forward propagation to obtain the actual output corresponding to the input data sample; obtain the loss gradient according to the preset loss function, the actual output and the label sample; update the weight matrix of the first self-attention layer according to the loss gradient using the backpropagation algorithm until the weight matrix meets the error requirement.

[0113] wherein each encoder has different training samples corresponding to it because of the different focus.

[0114] The traditional Transformer model uses a self-attention mechanism to capture the relationship within the sequence, and performs a weighted average on all key-value pairs without a clear way to distinguish which is key information and which is interference information. For ambiguous query intentions, the model may not accurately identify key information because it relies on a single attention mechanism to handle all types of semantic information. The embodiments of the present application can solve the above problems by introducing multiple attention encoders based on a multiple attention mechanism, each of which learns different representations of the input sequence. To implement the multiple attention mechanism, the embodiments of the present application pre-acquire corresponding training samples for each encoder, train the encoder using a preset loss function, and then obtain the weight matrix of each encoder. The different weight matrices obtained by training different training samples can focus on different information features during query sentence generation, further improving the accuracy of query sentence generation.

[0115] Optionally, the weight matrix includes self-attention weights, feedforward network weights, layer normalization weights, and position encoding weights.

[0116] In the embodiments of the present application, each encoder has an independent weight matrix, including self-attention weights, feedforward network weights, and layer normalization and position encoding weights. Each encoder can more accurately focus on different information.

[0117] S405: generating a target query sentence according to the context vectors corresponding to the encoders.

[0118] Here, the encoder of the embodiments of the present application includes a self-attention layer for assigning different attention weights to the target vector representation to extract semantic information, and a context vector layer for weighted processing to obtain a context vector. The self-attention layer is used to focus on different semantic features and accurately identify multiple key information in the query text. The context vector layer can output a context vector to help the large model focus on the relevant part of the encoder output, further improving the accuracy of query sentence generation.

[0119] Optionally, the embodiments of the present application can accurately generate a target query sentence based on the outputs of multiple encoders. Accordingly, FIG. 5 is a flowchart of another query sentence generation method based on a large language model provided by the embodiments of the present application, as shown in FIG. 5, the method includes:

[0120] S501: obtaining a query text.

[0121] S502: performing vectorization processing on the query text to obtain a target vector representation corresponding to the query text.

[0122] S503: input the target vector representation into a plurality of encoders to encode the target vector representation through the weight matrix of each encoder to obtain a context vector corresponding to each encoder.

[0123] wherein the weight matrix of different encoders is different.

[0124] wherein the implementation manner of steps S501-S503 is the same as that of steps S401-S404, which will not be repeated here.

[0125] S504: connecting the context vectors corresponding to each encoder to obtain a target context vector.

[0126] S505: generating a target query statement according to the target context vector.

[0127] Optionally, generating the target query statement according to the target context vector comprises: decoding the target context vector to obtain a structured query language template; and filling the structured query language template to obtain the target query statement.

[0128] wherein the embodiment of the present application first generates a structured query language template that accurately reflects the characteristics of the query statement based on the target context vector, and then obtains the target query statement by filling the template, thereby improving the accuracy and efficiency of query statement generation.

[0129] The embodiment of the present application connects the context vectors obtained by different encoders to obtain a target context vector that includes all context vectors, and generates a target query statement according to the target context vector. The target context vector includes information obtained by analyzing from different attention heads, so even if there is interference information, some attention heads can recognize and focus on key information, allowing the model to analyze and process query text from different angles. When processing ambiguous queries, different attention heads may capture different clues to help the model better understand the query intent, thereby improving the accuracy of query statement generation.

[0130] Optionally, the large language model further comprises a query intent recognizer, and the query intent recognizer is located in the decoder of the large language model; decoding the target context vector to obtain a structured query language template comprises: inputting the target context vector into the query intent recognizer to perform query intent recognition processing through the query intent recognizer to obtain at least one key information; and determining a structured query language label of the structured query language template according to the at least one key information.

[0131] Here, the embodiments of the present application set a query intent recognizer, which is a module specially used for identifying the user query intent, and it is crucial for generating accurate SQL templates. In the text-to-SQL task, the natural language query of the user can contain multiple intents, and the role of the query intent recognizer is to determine the query intent of the user, i.e., the key information, so as to generate a correct SQL statement. In the decoder, the query intent recognizer can help the model select the correct structure and keywords when generating the SQL template, thereby improving the accuracy of the query statement generation.

[0132] Optionally, the target context vector is input into the query intent recognizer to perform query intent recognition processing through the query intent recognizer to obtain at least one key information, including: inputting the target context vector into the query intent recognizer; in the query intent recognizer, calculating the target context vector according to a preset weight value, a preset bias value and a preset activation function in a preset intent recognition function, to obtain at least one key information.

[0133] Query intent recognizer: This is a module specially used for identifying the user query intent, and it is crucial for generating accurate SQL templates. In the text-to-SQL task, the natural language query of the user can contain multiple intents, such as "query" (SELECT), "update" (UPDATE), "insert" (INSERT) or "delete" (DELETE). The role of the query intent recognizer is to determine the query intent of the user, so as to generate a correct SQL statement. In the decoder, the query intent recognizer can help the model select the correct structure and keywords when generating the SQL template.

[0134] In a possible implementation, the working principle of the query intent recognizer is as follows: assuming that there is an input sequence Z of the decoder, which contains the context information output by the encoder. The goal of the query intent recognizer is to predict the intent y of the user query, and y can be one of multiple intent categories. The query intent recognizer can be represented as a function, as shown in the following formula (3): y=f(Z)=W T ·Relu(WZ+b) Formula (3)

[0135] In formula (3), W and b are the weight matrix and the bias of the query intent recognizer, and ReLU is the activation function. W and b can be determined according to actual conditions, and W Trepresents the dimension of the weight matrix W. The weight matrix W of the query intent recognizer can be learned through training so that the model can predict the correct query intent according to the features of the input sequence Z. During the decoding process, the output y of the query intent recognizer can be used to guide the decoder to generate the correct SQL template. For example, if the recognizer predicts the query intent as "query", the decoder will start generating an SQL template starting with "SELECT".

[0136] Optionally, the large language model further comprises a decoder module and a template guided attention module, the decoder module and the template guided attention module being located in a decoder of the large language model; the target context vector is subjected to decoding processing to obtain a structured query language template, comprising: inputting the target context vector into the decoder module for decoding processing, and obtaining a plurality of decoder outputs of the decoder module according to time steps; inputting the plurality of decoder outputs into the template guided attention module; in the template guided attention module, a predefined structured query language template is obtained; in the template guided attention module, the structure of the structured query language template is determined according to the predefined structured query language template and the plurality of decoder outputs.

[0137] Among them, the template guided attention module is specially designed to guide the Transformer model to follow a specific structure when generating SQL templates. This module can help the model generate structured SQL statements during the decoding process, especially when dealing with ambiguous queries. It can provide guidance when the decoder generates SQL statements, allowing the model to dynamically adjust the generated SQL template according to the input natural language query. It can effectively handle ambiguous queries and further improve the accuracy of query statement generation.

[0138] Optionally, in the template guided attention module, the structure of the structured query language template is determined according to the predefined structured query language template and the plurality of decoder outputs, comprising: in the template guided attention module, each decoder output is subjected to linear transformation processing to obtain a plurality of linear transformation outputs; in the template guided attention module, the predefined structured query language template is subjected to linear transformation processing to obtain a linear transformation template; in the template guided attention module, each linear transformation output and the linear transformation template are respectively subjected to calculation processing according to a pre-designed calculation function to obtain a plurality of attention outputs; in the template guided attention module, the plurality of attention outputs are subjected to splicing processing to obtain the structure of the structured query language template.

[0139] In one possible implementation, the template-guided attention module works as follows: the template-guided attention module can be designed as an attention mechanism that aligns the current output of the decoder with the corresponding part in the predefined SQL template. This module can be considered as an additional attention head that focuses on guiding the decoder to generate the correct SQL structure. Suppose we have an output h t of the decoder at time t n , and a predefined SQL template T = (t1, t2,..., t N ). The computation of the template-guided attention module can be represented as the following equation (4): TemplateAttention(h t , T) = tan(QK) · V Equation (4)

[0140] where in equation (4), Q is a linear transformation of the decoder output h t , K, V are linear transformations of the SQL template T. The output of the attention mechanism can be considered as the template-guided representation of the decoder at time t. This representation can be concatenated with other attention head outputs of the decoder or the output h t-1 at the previous time step to generate the final decoder output y t : y t = Decoder(TemplateAttention(h t , T); h t-1 ;...)

[0141] In this way, the template-guided attention module can dynamically provide guidance from the SQL template during the decoding process, helping the model to generate structured SQL statements. This mechanism is particularly useful for handling ambiguous queries, as it provides a structured framework that the decoder can fill in with specific query details.

[0142] Based on the above method, the current output of the decoder can be aligned with the corresponding part in the predefined SQL template, focusing on guiding the decoder to generate the correct SQL structure.

[0143] Optionally, the decoder module includes a second self-attention layer, a decoding attention layer, and a forward propagation module.

[0144] Optionally, at the decoder end, each decoder contains a context vector (also known as the output of "encoder-decoder attention"), which represents a weighted combination of the encoder output sequence. This context vector is used to generate the output of the next decoder. The context vector is a special self-attention mechanism in the decoder that helps the decoder focus on relevant parts of the encoder output.

[0145] Optionally, the decoder attention is part of the decoder that combines the self-attention mechanism and the context vector. Using the self-attention mechanism, attention is paid to previous tokens in the sequence that is currently being decoded (preventing future information leakage). Using the encoder-decoder attention mechanism, attention is paid to the context vector of the encoder output, ensuring that the output of the decoder is consistent with the context of the input sequence.

[0146] In step S503, the embodiment of the present application introduces a multi-attention layer in the encoder-decoder framework. The decoder focuses the multi-attention on the key information of the question encoding when generating each part of the SQL template. The multi-attention mechanism associates the question encoding with the current state of the decoder by calculating multi-attention weights. The multi-attention weights reflect the importance of each word in the question to the current decoder state. For example, keywords such as "list", "name", "email", etc. have special weights. Based on the multi-attention mechanism, an optional way of step S505 is described as follows: based on the multi-attention mechanism, the query intent recognizer, and the template guided attention module, the specific way of generating the query statement is as follows:

[0147] Step 505.1, calculate the multi-attention weight for the target context vector: according to the multi-attention weight, the decoder generates a specific SQL template. The multi-attention weight determines which question words will have a greater impact on the current decoder state. For example, when the decoder needs to generate a SELECT clause, "list" in the question may get a higher attention weight. When the decoder needs to generate a WHERE clause, "product X" in the question may get a higher attention weight. This process involves multiple encoders working together to identify key information in the user query and map it to different parts of the SQL template. In this way, the model can handle ambiguous queries and generate accurate SQL templates. This part can be performed by the query intent recognizer.

[0148] SELECT (SELECT): Attention may identify trigger words such as "select" or "query" and assign them high attention weights, and generate SQL templates according to the trigger words, for example:

[0149] Table name (FROM): One attention may focus on identifying table names.

[0150] Column (WHERE): One attention may focus on identifying table names.

[0151] Condition (=): One attention may focus on identifying table names.

[0152] Value: One attention can focus on identifying table names.

[0153] By combining the outputs of these attention heads, the decoder can gradually build up the SQL template

[0154] Step 505.2, Generating the key part of the SQL template: The decoder generates the key part of the SQL template based on the context vectors generated by the multi-attention mechanism. This part is generated by, for example, when generating the SELECT clause, the decoder focuses on the "user name and email address" in the question, and maps these information to the column names in the SQL template. The multi-attention weight determines which question words will have a greater impact on the current decoder state, thereby affecting the key part of the generated SQL template. The next SQL token is predicted using the current state of the decoder and the context vector calculated by the multi-attention mechanism, which can be executed by the template-guided attention module, and is specifically represented as follows: SQL_template = Decode(select_head, from_head, where_head, condition_head, value_head)

[0155] The decoder will output a series of SQL tokens, including SELECT, FROM, WHERE, etc.

[0156] The generated SQL template is as follows: SELECT <column1> ， <column2>FROM <column3>' value '

[0157] Optionally, after generating the SQL template, it is necessary to fill the template to generate diversified SQL queries. Accordingly, FIG. 6 is a flowchart of another method for generating a query statement based on a large language model according to an embodiment of the present application. As shown in FIG. 6, the method includes the following steps:

[0158] S601: Obtain a query text.

[0159] S602: Perform vectorization processing on the query text to obtain a target vector representation corresponding to the query text.

[0160] S603: Input the target vector representation into a plurality of encoders to perform encoding processing on the target vector representation through a weight matrix of each encoder to obtain a context vector corresponding to each encoder.

[0161] The weight matrix of different encoders is different.

[0162] S604: Perform connection processing on the context vector corresponding to each encoder to obtain a target context vector.

[0163] The implementation manners of steps S601-S604 are the same as those of steps S501-S504, and will not be repeated here.

[0164] S605: Perform decoding processing on the target context vector to obtain a structured query language template.

[0165] The implementation manner of step S605 can refer to the content of step S505 in the above embodiment, and will not be repeated here.

[0166] S606: Perform filling processing on the structured query language template to obtain a target query statement.

[0167] Optionally, the filling processing on the structured query language template to obtain the target query statement includes: identifying a placeholder in the structured query language template; obtaining a preset candidate set, wherein the preset candidate set includes a plurality of replacement items; and performing filling processing on the placeholder in the structured query language template according to the preset candidate set to obtain the target query statement.

[0168] Optionally, the filling processing on the placeholder in the structured query language template according to the preset candidate set to obtain the target query statement includes: performing multiple times of screening on the replacement items in the preset candidate set by using a bundle search algorithm, and filling the screened replacement items into the placeholder in the structured query language template until all the placeholders are replaced.

[0169] Wherein, the embodiment of the application fills the generated structured SQL template using the Beam Search algorithm to generate specific SQL queries. The Beam Search algorithm can maintain a certain "width" in the search space by retaining the most likely candidate results at each step, while limiting the "depth" of the search, to balance the breadth and depth of the search, reduce the amount of calculation, and improve the search efficiency. By combining the template generation based on the multiple attention mechanism and the template filling based on the Beam Search, the embodiment of the application can generate diversified SQL queries while maintaining the rationality of the query structure, improving the relevance and accuracy of the generated queries.

[0170] Optionally, the preset candidate set is obtained by querying the preset database to obtain the preset candidate set, and / or obtaining historical query data of a user corresponding to the query text, and generating the preset candidate set based on the historical query data.

[0171] Here, the embodiment of the application can establish the preset candidate set for filling the SQL template based on the preset database or the historical query data of the user corresponding to the query text, which can not only realize accurate SQL template filling through massive information in the preset database, but also realize customized SQL statement generation based on the user, further improving the flexibility and accuracy of the SQL statement generation and improving the user experience.

[0172] On the basis of the generated structured SQL template, the Beam Search algorithm is used for filling to generate specific SQL queries. The Beam Search algorithm can maintain a certain "width" in the search space by retaining the most likely candidate results at each step, while limiting the "depth" of the search, to balance the breadth and depth of the search, reduce the amount of calculation, and improve the search efficiency. By combining the template generation based on the multiple attention mechanism and the template filling based on the Beam Search, the application can generate diversified SQL queries while maintaining the legality of the query structure, improving the relevance and accuracy of the generated queries.

[0173] In one possible implementation, the steps of generating diversified SQL queries using template filling are as follows:

[0174] S606.1, Initialize Beam Search: Initialize the query generation process using the Beam Search algorithm. At each step, Beam Search keeps a pre-set number of candidate queries (e.g., top 5). Beam Search is a heuristic search algorithm mainly used in natural language processing (NLP) and machine translation for tasks that generate sequences, such as automatically generating text or translating sentences. The core idea is to keep a few most likely candidates at each step instead of just one optimal result, thus maintaining a certain "width" in the search space while limiting the "depth" of the search, balancing the breadth and depth of the search, reducing the computational load, and improving the search efficiency. Beam search searches over the probability distribution output by a generative model or decoder: given a starting symbol (e.g., a sentence start token or the beginning of an SQL query), the model predicts a probability distribution over the first element. Beam Search selects the top few elements (e.g., top 5), which are called "beams". For each candidate element, the model predicts a probability distribution over the next element. Then Beam Search selects the top element for each candidate element and adds these new elements to the corresponding candidate sequences. Beam Search updates the beams according to the probabilities of the new candidate sequences, keeping the top sequences. This process is repeated until the end condition of the sequence is reached (e.g., a sentence end token or the end of an SQL query). Beam Search outputs the top few sequences as the final result.

[0175] S606.2, Select and fill the first placeholder:

[0176] The first placeholder in the template is WHERE . According to the database schema, there is a whitelist containing table names, e.g., "users," "orders," "products." Beam Search considers all possible table names when filling in the

[0177] i

[0178] <column1> , <column2>FROM users WHERE <column3>= 'value'

[0179] S606.4, select and fill the second placeholder:

[0180] Now the need to fill in <column1>"and" <column2>". According to the schema of the "users" table, we have a whitelist of column names, e.g. "id", "name", "email". Beam Search in choosing <column1>"and" <column2>When filling in the blanks, all possible column names are considered, and the column name with the highest probability is selected. For example, "id" and "name" are likely to be the column names with the highest probability.

[0181] S606.5, Fill in the second placeholder:

[0182] Assuming "id" and "name" are the selected column names, the query template now looks like this:

[0183] SELECT id,name FROM users WHERE <column3>= 'value'

[0184] S606.6, select and fill the third placeholder:

[0185] Now the need to fill "" <column3>". According to the schema of the "users" table, continue using the whitelist of column names. Beam Search in choosing" <column3>At this point, all possible column names are considered, and the most probable column name is chosen for the fill. For example, "email" might be the most probable column name.

[0186] S606.7, fill the third placeholder: assuming "email" is the chosen column name, the final query template now looks like this: SELECT id, name FROM users WHERE email = 'value'

[0187] Through the above steps, a specific SQL query is generated using template filling. Throughout the process, Beam Search maintains multiple candidate queries and selects the most probable fill option at each step. This approach ensures that the generated query not only conforms to the template structure but is also based on the actual database schema, thereby improving the relevance and accuracy of the query.

[0188] Optionally, FIG. 7 is a structural diagram of another large language model according to an embodiment of the present application. As shown in FIG. 7, the large language model according to the embodiment of the present application includes two encoders, and each encoder includes two encoding layers. One linear function layer and two encoding layers constitute one encoder. The multiple attention mechanisms are realized through the two encoders, which can realize the functions of excluding interference words and supporting fuzzy queries in the text-to-sql task. Each encoding layer adopts a forward propagation mechanism and a self-attention mechanism. The decoder can include two decoding layers, a linear function layer, and a query intent recognizer. Each decoding layer adopts a template guide, a forward propagation, a decoding attention, and a self-attention mechanism. In order to further solve the irrelevant sentences in the model query and exclude inquiries, a query intent recognizer and a template guide attention module are also set. The query intent recognizer and the template guide attention module can be set in the decoder.

[0189] It should be noted that the structure of each encoder in FIG. 7 is the same. For example, the left encoder includes an encoding layer 1 and an encoding layer 2, and the right encoder includes an encoding layer 1 and an encoding layer 2. The same encoder 1 refers to the same structure and function of the two encoding layers. Both encoders include input embedding layers, and the functions and structures of the two input embedding layers are the same.

[0190] FIG. 8 is a structural schematic diagram of a query sentence generation device based on a large language model provided by an embodiment of the present application. As shown in FIG. 8, the device of the embodiment of the present application comprises an acquisition module 801, a first processing module 802, a second processing module 803 and a generation module 804. The query sentence generation device based on a large language model herein can be the processing device itself or a chip or integrated circuit implementing the function of the processing device. It should be noted that the division of the acquisition module 801, the first processing module 802, the second processing module 803 and the generation module 804 is only a logical division, and physically, the two can be integrated or independent.

[0191] The acquisition module is configured to acquire a query text.

[0192] The first processing module is configured to perform vectorization processing on the query text to obtain a target vector representation corresponding to the query text.

[0193] The second processing module is configured to input the target vector representation into a plurality of encoders to perform encoding processing on the target vector representation through a weight matrix of each encoder to obtain a context vector corresponding to each encoder, wherein the weight matrix of different encoders is different.

[0194] The generation module is configured to generate a target query sentence according to the context vector corresponding to each encoder.

[0195] Optionally, the encoder comprises a first self-attention layer and a context vector layer.

[0196] The second processing module is specifically configured to input the target vector representation into the first self-attention layer of the plurality of encoders to calculate the target vector representation through the weight matrix of the first self-attention layer of each encoder to obtain a plurality of output vectors of each encoder; and input the plurality of output vectors of each encoder into the corresponding context vector layer to perform weighted processing on the plurality of output vectors through each context vector layer to obtain the context vector corresponding to each encoder.

[0197] Optionally, before the second processing module is configured to input the target vector representation into the first self-attention layer of the plurality of encoders to calculate the target vector representation through the weight matrix of the first self-attention layer of each encoder to obtain a plurality of output vectors of each encoder, the device further comprises a training module configured to: for each encoder, acquire a training sample corresponding to the encoder, wherein the training sample comprises an input data sample and a label sample; input the training sample into the first self-attention layer for forward propagation to obtain an actual output corresponding to the input data sample; obtain a loss gradient according to a preset loss function, the actual output and the label sample; and update the weight matrix of the first self-attention layer according to the loss gradient by using a back propagation algorithm until the weight matrix meets an error requirement.

[0198] Optionally, the weight matrix comprises self-attention weights, feedforward network weights, layer normalization weights, and position encoding weights.

[0199] Optionally, the generation module comprises:

[0200] a concatenation sub-module configured to concatenate the context vectors corresponding to the encoders to obtain a target context vector;

[0201] a generation sub-module configured to generate the target query statement according to the target context vector.

[0202] Optionally, the generation sub-module further comprises:

[0203] a decoding sub-module configured to decode the target context vector to obtain a structured query language template;

[0204] a filling sub-module configured to fill the structured query language template to obtain the target query statement.

[0205] Optionally, the large language model further comprises a query intent recognizer, the query intent recognizer being located in a decoder of the large language model; and the decoding sub-module is specifically configured to: input the target context vector into the query intent recognizer to perform query intent recognition processing through the query intent recognizer to obtain at least one key information; and determine a structured query language token of the structured query language template according to the at least one key information.

[0206] Optionally, the decoding sub-module is further specifically configured to: input the target context vector into the query intent recognizer; and in the query intent recognizer, calculate the target context vector according to a preset weight value, a preset bias value, and a preset activation function in a preset intent recognition function to obtain the at least one key information.

[0207] Optionally, the large language model further comprises a decoder module and a template-guided attention module, the decoder module and the template-guided attention module being located in a decoder of the large language model; and the decoding sub-module is specifically configured to:

[0208] input the target context vector into the decoder module for decoding processing, and obtain a plurality of decoder outputs of the decoder module according to a time step; input the plurality of decoder outputs into the template-guided attention module; in the template-guided attention module, obtain a predefined structured query language template; and in the template-guided attention module, determine a structure of the structured query language template according to the predefined structured query language template and the plurality of decoder outputs.

[0209] Optionally, the decoding submodule is further configured to perform linear transformation processing on each decoder output in the template-guided attention module to obtain a plurality of linear transformation outputs; perform linear transformation processing on the predefined structured query language template in the template-guided attention module to obtain a linear transformation template; perform calculation processing on each linear transformation output and the linear transformation template according to a pre-designed calculation function in the template-guided attention module to obtain a plurality of attention outputs; and perform splicing processing on the plurality of attention outputs in the template-guided attention module to obtain the structure of the structured query language template.

[0210] Optionally, the decoder module comprises a second self-attention layer, a decoding attention layer and a forward propagation module.

[0211] Optionally, the padding submodule is configured to identify the placeholder in the structured query language template; and obtain a preset candidate set, wherein the preset candidate set comprises a plurality of replacement items.

[0212] The placeholder in the structured query language template is filled according to the preset candidate set to obtain the target query sentence.

[0213] Optionally, the padding submodule is further configured to use a beam search algorithm to perform multiple screenings on the replacement items in the preset candidate set, and fill the screened replacement items into the placeholder in the structured query language template until all the placeholders are replaced.

[0214] Optionally, the obtaining module is configured to query the preset database to obtain the preset candidate set; and / or obtain historical query data of a user corresponding to the query text, and generate the preset candidate set according to the historical query data.

[0215] Optionally, the first processing module is configured to input the query text into the pre-trained language model to obtain the target vector representation corresponding to the query text through the output of the pre-trained language model.

[0216] Optionally, the large language model further comprises an input embedding layer and a first linear function layer; and the first processing module is further configured to input the query text into the input embedding layer to output an embedding vector corresponding to the query text through the input embedding layer; and input the embedding vector into the first linear function layer to output the target vector representation corresponding to the query text through the first linear function layer.

[0217] Referring to FIG. 9, a structural diagram of a large language model-based query sentence generation device 900 suitable for implementing embodiments of the disclosure is illustrated, which can be a terminal device or a server. The terminal device can include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a personal digital assistant (PDA), a tablet computer (PAD), a portable multimedia player (PMP), a vehicle-mounted terminal (e.g., a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. The large language model-based query sentence generation device illustrated in FIG. 9 is merely an example, and should not impose any limitation on the functions and use range of embodiments of the disclosure.

[0218] As illustrated in FIG. 9, the large language model-based query sentence generation device 900 can include a processing device (e.g., a central processing unit, a graphic processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the large language model-based query sentence generation device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0219] Generally, the following devices can be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; a storage device 908 including, for example, a magnetic tape, a hard disk, and the like; and a communication device 909. The communication device 909 can allow the large language model-based query sentence generation device 900 to communicate with other devices wirelessly or by wire to exchange data. Although FIG. 9 illustrates the large language model-based query sentence generation device 900 having various devices, it should be understood that all of the illustrated devices are not required to be implemented or provided. More or fewer devices can be alternatively implemented or provided.

[0220] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication device 909, or installed from the storage device 908, or installed from the ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0221] It should be noted that the computer readable medium described above in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can transmit, propagate or transport program for use by or in connection with an instruction execution system, device or apparatus. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, cable, RF (radio frequency), or the like, or any suitable combination of the above.

[0222] The above computer readable medium can be contained in the above described large language model based query sentence generation device; or can exist separately and not be assembled into the large language model based query sentence generation device.

[0223] The computer readable medium described above carries one or more programs, which when executed by the large language model based query statement generation device, cause the large language model based query statement generation device to perform the method shown in the above embodiment.

[0224] Computer program code for carrying out operations of the present disclosure can be written in one or more programming languages or combinations of languages including object oriented programming languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0225] The flow diagrams and the block diagrams in the drawings are illustrations of possible architectures, functions, and operations for systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0226] The units described in the embodiments of the present disclosure can be implemented by software, or by hardware. In some cases, the name of the unit does not constitute a limitation on the unit itself. For example, the first obtaining unit can also be described as a unit for obtaining at least two Internet protocol addresses.

[0227] The functionality described herein above can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc.

[0228] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0229] The query sentence generation device based on a large language model provided in the embodiments of the present application can be used to execute the technical solutions in the method embodiments of the present application, and the implementation principles and technical effects are similar, which will not be described here.

[0230] The embodiments of the present application also provide a computer readable storage medium, which stores computer execution instructions. When the computer execution instructions are executed by a processor, the computer execution instructions are used to implement the query sentence generation method based on a large language model of any one of the above.

[0231] The embodiments of the present application also provide a computer program product, which includes a computer program. When the computer program is executed by a processor, the computer program is used to implement the query sentence generation method based on a large language model of any one of the above.

[0232] In several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, and the division of units is merely a logical function division. In actual implementation, another division manner can be adopted, for example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms. In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can be physically present separately, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0233] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0234] It will be understood that the application is not limited to the precise structures hereinbefore described and illustrated in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the application is indicated by the appended claims. < / column1> , and selects the table name with the highest probability as the fill-in. For example, "users" can be the table name with the highest probability. The probability calculation formula is as formula (5): In formula (5), represents the probability calculation result, Q is the question, D is the database, L(D) is the description of D, P is the prompt word, s is the query question, and s is the i-th token of the query question. S606.3, Fill in the first placeholder: Assuming that "users" is the selected table name, the query template now looks like this: SELECT < / column1>

Claims

1. A method for generating a query sentence based on a large language model, wherein, The large language model comprises a plurality of encoders, and the method comprises: obtaining query text; vectorizing the query text to obtain a target vector representation corresponding to the query text; inputting the target vector representation into the plurality of encoders to encode the target vector representation through the weight matrix of each encoder to obtain a context vector corresponding to each encoder, wherein the weight matrix of different encoders is different; generating a target query statement according to the context vector corresponding to each encoder.

2. The method of claim 1, wherein, The encoder comprises a first self-attention layer and a context vector layer; The inputting the target vector representation into the plurality of encoders to encode the target vector representation through the weight matrix of each encoder to obtain a context vector corresponding to each encoder comprises: inputting the target vector representation into the first self-attention layer of each encoder to calculate the target vector representation through the weight matrix of the first self-attention layer of each encoder to obtain a plurality of output vectors of each encoder; inputting the plurality of output vectors of each encoder into the corresponding context vector layer to weight the plurality of output vectors through each context vector layer to obtain a context vector corresponding to each encoder.

3. The method of claim 1 or 2, wherein, Before the inputting the target vector representation into the first self-attention layer of each encoder to calculate the target vector representation through the weight matrix of the first self-attention layer of each encoder to obtain a plurality of output vectors of each encoder, the method further comprises: for each encoder, obtaining a training sample corresponding to the encoder, wherein the training sample comprises an input data sample and a label sample; forward propagating the training sample through the first self-attention layer to obtain an actual output corresponding to the input data sample; obtaining a loss gradient according to a preset loss function, the actual output and the label sample; updating the weight matrix of the first self-attention layer according to the loss gradient by using a back propagation algorithm until the weight matrix meets an error requirement.

4. The method according to any one of claims 1 to 3, wherein, The weight matrix comprises a self-attention weight, a feedforward network weight, a layer normalization weight and a position encoding weight.

5. The method according to any one of claims 1 to 4, wherein, The generating a target query statement according to the context vector corresponding to each encoder comprises: connecting the context vectors corresponding to the encoders to obtain a target context vector; generating a target query statement according to the target context vector.

6. The method of claim 5, wherein, The generating a target query statement according to the target context vector comprises: decoding the target context vector to obtain a structured query language template; filling the structured query language template to obtain a target query statement.

7. The method of claim 6, wherein, The large language model further comprises a query intent recognizer, and the query intent recognizer is located in a decoder of the large language model; The decoding the target context vector to obtain a structured query language template comprises: inputting the target context vector into the query intent recognizer to perform query intent recognition processing through the query intent recognizer to obtain at least one key information; According to the at least one key information, a structure of the structured query language template is determined.

8. The method of claim 7, wherein, The target context vector is input into the query intent recognizer, and query intent recognition processing is performed on the target context vector by the query intent recognizer to obtain at least one key information. The target context vector is input into the query intent recognizer. In the query intent recognizer, the target context vector is calculated according to a preset weight value, a preset bias value and a preset activation function in a preset intent recognition function to obtain at least one key information.

9. The method according to any one of claims 6-8, wherein, The large language model further comprises a decoder module and a template guided attention module, and the decoder module and the template guided attention module are located in a decoder of the large language model. The target context vector is decoded to obtain a structured query language template, comprising: The target context vector is input into the decoder module for decoding processing, and a plurality of decoder outputs of the decoder module are obtained according to time steps. The plurality of decoder outputs are input into the template guided attention module. In the template guided attention module, a predefined structured query language template is obtained. In the template guided attention module, a structure of the structured query language template is determined according to the predefined structured query language template and the plurality of decoder outputs.

10. The method of claim 9, wherein, In the template guided attention module, a structure of the structured query language template is determined according to the predefined structured query language template and the plurality of decoder outputs, comprising: In the template guided attention module, each of the decoder outputs is linearly transformed to obtain a plurality of linearly transformed outputs. In the template guided attention module, the predefined structured query language template is linearly transformed to obtain a linearly transformed template. In the template guided attention module, each of the linearly transformed outputs and the linearly transformed template is calculated according to a preset calculation function to obtain a plurality of attention outputs. In the template guided attention module, the plurality of attention outputs are spliced to obtain the structure of the structured query language template.

11. The method of claim 9 or 10, wherein, The decoder module comprises a second self-attention layer, a decoding attention layer and a forward propagation module.

12. The method according to any one of claims 6 to 11, wherein, The structured query language template is filled to obtain a target query sentence, comprising: The placeholder in the structured query language template is identified. A preset candidate set is obtained, wherein the preset candidate set comprises a plurality of replacement items. The placeholder in the structured query language template is filled according to the preset candidate set to obtain a target query sentence.

13. The method of claim 12, wherein, The placeholder in the structured query language template is filled according to the preset candidate set to obtain a target query sentence, comprising: A bundle search algorithm is used to screen the replacement items in the preset candidate set multiple times, and the screened replacement items are filled into the placeholder in the structured query language template until all the placeholders are replaced.

14. The method of claim 12, wherein, The obtaining of the preset candidate set comprises: querying in a preset database to obtain the preset candidate set; and / or, obtaining historical query data of a user corresponding to the query text, and generating a preset candidate set according to the historical query data.

15. The method according to any one of claims 1 to 14, wherein, The vectorization processing of the query text to obtain the target vector representation corresponding to the query text comprises: inputting the query text into a pre-trained language model to obtain the target vector representation corresponding to the query text through the output of the pre-trained language model.

16. The method of any one of claims 1 to 15, wherein, The large language model further comprises an input embedding layer and a first linear function layer. The vectorization processing of the query text to obtain the target vector representation corresponding to the query text comprises: inputting the query text into the input embedding layer to output an embedding vector corresponding to the query text through the input embedding layer; inputting the embedding vector into the first linear function layer to output the target vector representation corresponding to the query text through the first linear function layer.

17. A large language model-based query sentence generation device, wherein, comprise: a processor, and a memory connected in communication with the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to realize the method in any one of claims 1 to 16.

18. A computer readable storage medium, wherein, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to realize the method in any one of claims 1 to 16.

Images