Example library construction method and device, query method and device, and large language model application system

Abstract

The application provides an example library construction method and device, a query method and device, and a large language model application system. The example library construction method comprises: receiving a first example; expanding the first example according to a meta template and a first large language model to generate a second example; and generating an example library according to the second example. The example library is used to provide the second example for a second large language model. The second example is an injection example of the second large language model, and the second example matches a user query. The second large language model is used to provide an answer for the user query. The method can expand a plurality of examples from a small number of examples provided by a user according to a meta template and a large language model, and construct an example library, thereby improving the efficiency of example generation, increasing the coverage of examples, providing rich retrieval example resources for user queries, improving the effectiveness of examples injected into prompt words, and enhancing the repair result of a fault case.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to methods and apparatus for constructing example libraries, methods and apparatus for querying, and application systems for large language models. Background Technology

[0002] With the continuous improvement of Large Language Model (LLM) capabilities, applications centered around LLM are constantly emerging. Developers can create personalized LLM applications through web page configuration and publish them on the platform for other users to use.

[0003] Applications that rely on large language models inevitably face the problem of bad cases. A common approach to fixing bad cases in large language model applications is to construct labeled samples based on online feedback of bad cases to generate a preference training set, and then fine-tune the model's parameters based on this set. However, due to the high training cost and long training cycle of large language models, frequent parameter fine-tuning cannot be used to fix bad cases. Furthermore, parameter fine-tuning requires the model to be open-source; for applications using non-open-source models, fixing bad cases through parameter fine-tuning is not feasible.

[0004] Besides parameter fine-tuning, example enhancement can also be used to repair faulty use cases. Example enhancement involves injecting examples into prompts, guiding the large language model to respond in the expected way based on the injected examples. However, as the number of faulty use cases increases, not only is manual processing inefficient and lacking in coverage, but the number of examples that need to be injected into the prompts increases linearly, eventually exceeding the length limit of the prompts, affecting the effectiveness of the prompts, and thus limiting the repair results of faulty use cases. Summary of the Invention

[0005] This application provides a method and apparatus for building an example library, a query method and apparatus, and a large language model application system. It can expand a small number of examples provided to users based on meta-templates and large language models to generate multiple examples and build an example library. This improves the efficiency of example generation, increases the coverage of examples, and provides rich retrieval example resources for user queries to ensure that examples matching the user's query can be retrieved. This improves the effectiveness of examples injected into prompt words and enhances the repair results of fault test cases.

[0006] In a first aspect, this application provides a method for constructing an example library, comprising: receiving a first example; inputting a meta-template and the first example into a first large language model to generate a second example; both the first example and the second example include a question and an answer; the number of second examples is greater than the number of first examples; generating an example library based on the second examples; the example library is used to provide injected examples for the second large language model, and the injected examples are second examples related to user queries, and the second large language model is used to provide answers to user queries.

[0007] The example library construction method provided in this application can expand a small number of examples provided to users based on meta-templates and large language models to create multiple examples and build an example library, thereby improving the efficiency of example generation while increasing the coverage of examples. This example library can provide rich retrieval example resources for user queries, ensuring that examples matching the user's query can be retrieved, thus improving the effectiveness of examples injected into prompts and enhancing the repair results of faulty use cases.

[0008] One possible implementation involves inputting a meta-template and a first example into a first large language model to generate a second example. This includes: determining a topic related to the first example; constructing a first template based on the topic, the meta-template, and the first example related to the topic, and inputting the first template into the first large language model to obtain a second template; and constructing a third template based on the second template, and inputting the third template into the first large language model to obtain a second example. In this approach, a second template is constructed for each topic using the large language model, and multiple second examples are generated for each second template based on the large language model, thus enabling the generation of multiple second examples for each topic according to the large language model. Therefore, this application can expand a small number of examples while adhering to the generation rules of the large language model, and can guarantee the coverage of the expanded examples.

[0009] One possible implementation involves performing quality checks on the second examples based on the first major language model, removing invalid examples. Invalid examples are those that do not meet the requirements of reasonableness, format consistency, and / or diversity. In this approach, each second example is quality-checked according to three dimensions: reasonableness, format consistency, and diversity, to remove invalid examples. The remaining second examples are then used to build an example library, thereby improving the effectiveness of the second examples in the library.

[0010] One possible implementation involves constructing a vector engine for the example library based on an expert model. In this approach, each second example undergoes quality checks based on three dimensions: reasonableness, format consistency, and diversity. Invalid examples are removed, and the remaining second examples are used to construct the example library, thereby improving the effectiveness of the second examples within it. This method also improves the accuracy of example retrieval by introducing an expert model to build the vector engine.

[0011] Secondly, this application provides a query method, including: receiving a user query; retrieving multiple second examples from an example library based on the user query; injecting the multiple second examples into a prompt word template to generate prompt words; and inputting the prompt words and the user query into a second large language model to output the answer corresponding to the user query.

[0012] The query method provided in this application injects examples retrieved from an example library that match the user's query into prompt words to generate a response corresponding to the user's query. Since the examples retrieved from the example library are those that match the user's query, it not only effectively controls the number of examples injected into the prompt words but also improves the effectiveness of the examples injected, thereby enhancing the repair results of faulty use cases.

[0013] One possible implementation involves, after injecting multiple second examples into the prompt word template to generate prompt words, adjusting the positional distribution of these second examples within the prompt words based on the user query. For a user query, if there are multiple second examples, their positional distribution affects the effectiveness of the generated answer; different user queries have different requirements for the positional distribution of the second examples. Therefore, dynamically adjusting the positional distribution of the second examples based on the user query generates the optimal second examples suitable for the user query, thereby generating the optimal prompt words. Compared to the case where the positional distribution of the examples injected into the prompt words remains static, dynamically adjusting the positional distribution of each example injected into the prompt words based on the user query generates the optimal example positional distribution suitable for the user query, improving the effectiveness of the generated prompt words and thus improving the repair effect of faulty test cases.

[0014] As one possible implementation, the prompt word includes multiple elements; after injecting the second example into the prompt word template to generate the prompt word, the method further includes: adjusting the positional distribution of multiple elements in the prompt word according to the user query. In this method, the positional distribution of elements affects the effectiveness of the generated answer for the user query, and different user queries have different requirements for the positional distribution of elements. Therefore, after the content of each element is filled in, the element sorter adjusts the position of the elements in the prompt word template based on the user query to generate the optimal prompt word suitable for the user query, and thus generate the optimal answer. Here, the positional distribution of elements refers to the positional distribution of each element in the prompt word. Compared with the case where the element sorting remains static, the positional distribution of each element in the prompt word can be dynamically adjusted according to the user query to generate the optimal element positional distribution suitable for the user query, thereby improving the effectiveness of the generated prompt word and thus improving the repair effect of fault test cases.

[0015] As one possible implementation, the elements include role, objective, skill, example, and limitation. In this approach, the elements included in the prompt are specifically defined.

[0016] One possible implementation involves, after injecting the second example and the user query into the prompt word template to generate prompt words, the following steps are taken: vectorizing the user query; determining the weights of multiple experts based on the vectorized representation of the user query; having the multiple experts search for a second example matching the user query from an example library; calculating the score of the second example matching the user query based on the weights; and determining the second example from the second examples matching the user query based on the score. In this approach, example retrieval is performed using an expert model, thus improving the accuracy of example retrieval.

[0017] Thirdly, this application provides an example library construction apparatus for executing the methods in the first aspect or any possible implementation of the first aspect. Specifically, the apparatus includes modules for executing the methods in the first aspect or any possible implementation of the first aspect.

[0018] Fourthly, this application provides a query apparatus for executing the method in any possible implementation of the second aspect or the first aspect. Specifically, the apparatus includes a module for executing the method in any possible implementation of the second aspect or the first aspect.

[0019] Fifthly, this application provides a large language model application system, including the apparatus of the third and fourth aspects.

[0020] In a sixth aspect, this application provides an electronic device including a processor and a memory interconnected thereto, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is used to invoke the program instructions to execute the first aspect, the second aspect, any possible implementation of the first aspect, or a possible implementation of the second aspect.

[0021] In a seventh aspect, this application provides a computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it performs the first aspect, the second aspect, any possible implementation of the first aspect, or a possible implementation of the second aspect.

[0022] Eighthly, this application provides a computer program product having a computer program stored thereon, wherein when the computer program is executed by a processor, it performs the first aspect, the second aspect, any possible implementation of the first aspect, or a possible implementation of the second aspect. Attached Figure Description

[0023] Figure 1 This is an example diagram of an existing repair method;

[0024] Figure 2 Here is an example diagram of another existing example repair method;

[0025] Figure 3 This is a schematic diagram of the system framework provided in this application;

[0026] Figure 4 This is an example diagram of a large language model application system provided in this application;

[0027] Figure 5 This is another example diagram of a large language model application system provided in this application;

[0028] Figure 6 This is a functional block diagram of the example library construction apparatus provided in this application;

[0029] Figure 7 These are extended example diagrams provided in this application;

[0030] Figure 8 This is a diagram of the sub-functional modules of the example extended module provided in this application;

[0031] Figure 9 This is a functional module diagram of the query device provided in this application;

[0032] Figure 10 This is an example diagram of expert model retrieval provided in this application;

[0033] Figure 11 This is an example diagram of the prompt word generation process provided in this application;

[0034] Figure 12 This is a flowchart of the sample library construction method provided in this application;

[0035] Figure 13 This is a flowchart of the example extended method provided in this application;

[0036] Figure 14 This is a flowchart of the query method provided in this application;

[0037] Figure 15 Example diagram of a computing device according to an embodiment of this application. Detailed Implementation

[0038] The embodiments of this application will now be described with reference to the figures. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0039] The terms "first," "second," etc., in the specification, claims, and figures of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0040] To facilitate understanding of this application, the terms used in this application will be explained first.

[0041] (1) Large Language Model

[0042] Large Language Models (LLMs) are machine learning models trained on large amounts of text data. They can be used to generate natural language text or understand the meaning of language text. LLMs can handle various natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence. Currently, LLMs employ a Transformer architecture and pre-training objectives similar to small models. The difference between large and small models lies in the increased model size, training data, and computational resources.

[0043] (2) Example

[0044] The example package contains questions and answers to guide the large language model on how to respond to new questions. Because the large language model has semantic generalization capabilities, it is expected to respond to new questions similar to those in the examples in a similar format and style.

[0045] (3) Fault Cases

[0046] A bad base is an example identified by the user that does not meet expectations during the process of providing services to the user.

[0047] (4) Prompt words

[0048] Prompts are instructions or phrases used to interact with a large language model, specifying what tasks the model should perform and what output it should generate. Prompts act as a bridge between the user and the large language model, enabling the model to understand and respond accordingly or generate creative content. By designing and optimizing prompts, large language models can be utilized more effectively to accomplish various tasks.

[0049] This application's embodiments relate to the repair of faulty use cases in large language model application scenarios. Traditional faulty use case repair includes two methods: parameter fine-tuning and example enhancement. (See also...) Figure 1 In the fault case repair method based on parameter fine-tuning, fault cases are collected based on users' online feedback. These collected fault cases are used as samples and input into the large language model to obtain two different response results. The response results are labeled using an annotation model. Positive and negative samples are generated based on the labeled response results and the corresponding questions. A preference training set is constructed based on the positive and negative samples. The parameters of the large language model are fine-tuned based on the constructed preference training set.

[0050] Fault test case repair methods based on parameter fine-tuning are only applicable to open-source models. However, many large-scale model applications are not open-source. For example, many applications rely heavily on OpenAI's GPT-4 model. For these applications based on non-open-source models, using parameter fine-tuning to repair fault test cases is not feasible. Furthermore, due to the large parameter scale of large language models, parameter fine-tuning requires significant GPU resources and the training process is time-consuming. Relying on parameter fine-tuning cannot achieve immediate repair of fault test cases, which can easily lead to user churn.

[0051] See Figure 2 In the example-enhanced fault case repair method, expected example responses are manually written based on fault cases reported online by users, generating example question-and-answer pairs. When users use a large language model for text generation, these example question-and-answer pairs are injected into prompts to guide the large language model to generate responses that more closely match the user's expectations. The number of example question-and-answer pairs can be one or more.

[0052] In the example-enhanced fault test case repair method, example question-answer pairs need to be constructed manually, which has low processing efficiency, limited coverage of example question-answer pairs, and the number of examples that need to be injected into the prompt words increases linearly, eventually exceeding the length limit of the prompt words, affecting the effectiveness of the prompt words, and thus limiting the repair results of fault test cases.

[0053] In view of this, embodiments of this application provide a method for constructing an example library, a query method, and a large language model application system. The technical solutions provided by this application will be described below.

[0054] Figure 3This is an example diagram of the system framework provided in this application embodiment, including software and hardware components. The software component includes a machine learning platform 300, which includes an example expansion service 301, a vector engine 302, an example retrieval service 303, a prompt word generation service 304, and a large language model service 305. Except for the large language model service 305, the others are services or modules newly added to the machine learning platform in this application embodiment compared to existing technologies. The hardware component includes a processor 306 and a memory 307. The memory 307 stores an example library 308, which stores multiple examples, each including a question and an answer. The machine learning platform 300 provides an input interface for first examples. The number of first examples is small and insufficient to cover multiple topics to complete the construction of the example library. Each first example is a question-answer pair, including a question and an answer. The example expansion service 301 is used to expand the number and topics of the small number of first examples input by the user to generate second examples with broader coverage. The vector engine 302 is used to build an index for the example library 308 based on the second examples. Example extended service 301 and vector engine 302 can be integrated into the same service unit or exist independently.

[0055] The machine learning platform 300 provides a user query interface. The example retrieval service 303 uses this interface to find a second example from the example library that matches the user query. The prompt word generation service 304 injects the second example into the prompt word template to generate prompt words. The large language model service 305 provides a computational model for building the example library and generating the second example. Because the second example injected into the prompt words is relevant to the user query, it not only effectively controls the number of examples injected into the prompt words but also improves the effectiveness of the examples injected.

[0056] The large language model in this application can be any large language model that can be applied in the prior art. It can be an open source model or a non-open source model, such as the GPT series, Coze, PaLM, ChatGPT, Pangu large model, etc. It should be noted that this is not an exhaustive list.

[0057] The memory 307 stores the example library 308 and program code deployed on the hardware. The program code in this embodiment runs in the processor 306 to implement the functions of each service / module. The hardware of this application can be deployed on the same electronic device as the software, or on different electronic devices. All software services can be deployed on the same electronic device, or services / modules can be deployed on different electronic devices. All hardware functions can be deployed on the same electronic device, or services / modules can be deployed on different electronic devices.

[0058] Figure 4 This is an example diagram of a large language model application system provided in this application embodiment, which includes five parts: example import, example expansion, example library, example retrieval, example optimization, and input and output of the large language model. In this example scenario, after importing a small number of initial examples, the initial examples are expanded, and an example library is generated based on the expanded examples. When a user query request is received, the example library is retrieved according to the user query. The retrieved examples are filtered and sorted according to the example sorter and feature sorter to obtain optimized examples. The optimized examples and the user query are injected into prompt words. The user query and the injected optimized example prompt words are input into the large language model to generate an answer corresponding to the user query. The process of generating the example library can be an offline process, while the process of generating the answer corresponding to the user query can be an online process.

[0059] The large language model application system provided in this application embodiment can be divided into an offline stage and an online stage. The offline stage is used to generate an example library, and the online stage is used to apply the large language model using the example library. (See also...) Figure 5 The offline phase includes processes such as example import, example expansion, and example library construction. The online phase includes processes such as example retrieval, example optimization, and generating answers corresponding to user queries based on the large language model. Among these, example retrieval is the process of retrieving examples related to user queries from the example library.

[0060] See Figure 6 This application provides an example library building apparatus 600, which includes:

[0061] The first receiving module 601 is used to receive the first example.

[0062] The first example is the initial example input by the user. There may be one or more first examples. Compared to the expanded second examples, the first example input by the user is a small number of examples. For example, if the number of second examples is 200 and the number of first examples is 5, the first example is a small number of examples relative to the second example. It should be noted that the number here is only an example and is not intended to limit this application.

[0063] Users import first examples in batches using the input interface provided by the machine learning platform. The first examples can be user-annotated question-and-answer pairs or unannotated question-and-answer pairs; they can be online feedback question-and-answer pairs, user-constructed question-and-answer pairs, or the union of online feedback question-and-answer pairs and user-constructed question-and-answer pairs (excluding online feedback pairs). After the user imports the first examples, the platform receives them, for example... Figure 7 The image shown is a first example of user input regarding a travel plan to Beijing. Figure 7In this example, the sample input represents the question, and the sample output represents the answer. Each first example is a complete question-and-answer pair.

[0064] The second example generation module 602 is used to input the meta template and the first example into the first large language model to generate the second example.

[0065] Each second example is a complete question-and-answer pair. The purpose of expanding the first examples is to increase the number and topics of the few first examples provided by the user input, so as to improve the efficiency of second example generation and increase the coverage of the second examples.

[0066] Reference Figure 8 The second example generation module 602 includes a topic determination module 6021, a second template generation module 6022, and an example expansion module 6023. The functions of each module are described below.

[0067] The topic determination module 6021 is used to determine the topic related to the first example.

[0068] Topics covering multiple dimensions can be pre-defined, and these topics can include a single level or multiple levels. The lowest level topic matching the first example can be designated as related to the first example, or other levels of topics can be designated as related, or all topics within the same level of the matched topic can be designated as related to the first example. The choice of which level to use as related topics can be determined by the number of first examples entered by the user and the number of matched topics, or by the platform's default settings, to avoid having too many or too few topics identified.

[0069] There can be one or more topics associated with a first example. In this embodiment, the determined topics associated with a first example are the sum of all topics associated with the first example, and the number can be one or more. In this embodiment, it is assumed that there are T topics associated with the first example, where T is greater than or equal to 1.

[0070] For multi-level themes, assuming that the first-level themes include tourism, technology, medicine, philosophy, etc., and the second-level themes include domestic tourism, European tourism, African tourism, South Korean tourism, etc., then... Figure 3 In the first example shown, assuming the number of first examples entered by the user is 1, the topics related to the first example are determined by matching with the set topics, such as domestic tourism, European tourism, and African tourism.

[0071] The second template generation module 6022 is used to construct a first template based on the topic, meta template and a first example related to the topic, and input the first template into the first large language model to obtain the second template.

[0072] Meta-templates are used to guide the first language model to generate different outputs based on different inputs; they are preset templates. During the generation of the second template, meta-templates are used to construct different first templates suitable for the first language model based on different topics and different first examples. This guides the first language model to generate different second templates based on the different first templates, ensuring the diversity and effectiveness of the generated second templates.

[0073] For a given theme, when constructing the first template, a meta template is first obtained. Then, the first template is constructed based on the theme, the first examples related to the theme, and the meta template. Multiple meta templates can be set. The meta template used to construct the first template can be selected based on the first examples or can be a default meta template. The number of first examples related to a theme can be one or more. A first template can contain the original content of the first examples under the corresponding theme or it can contain modified first example content. One first template is constructed for each theme, and one second template is generated from each first template; that is, the number of second templates is also T.

[0074] The generated second template contains two parts: a question and an answer. The question and answer in the second template can be the same as or different from the corresponding first example; the content of each second template is unique. For example, for... Figure 7 In the first example shown, each of the output second templates contains two parts: a question and an answer. The questions in the different second templates are "Please design a travel plan for me to go to Shanghai", "Please design a travel plan for me to go to Rome", "Please design a travel plan for me to go to Africa", etc.

[0075] The second example generation module 6023 is used to construct a third template based on the second template and input the third template into the first large language model to obtain the second example.

[0076] A third template is constructed for a second template. The third template may be the same as or different from the corresponding second template. For a single third template, the number of second examples generated is multiple, and different third templates generate the same number of second examples. In this embodiment, it is assumed that the number of second examples corresponding to a third template is M, where M is greater than 1. The third template and the first template may be the same or different.

[0077] It should be noted that the expanded examples may or may not include the first example, which can be controlled by the construction rules of the first and third templates. Each second example is a question-and-answer pair, containing a question and an answer. The content of the second example can be the same as or different from the second template.

[0078] exist Figure 7 In the second example shown, for the second template with the question "Please design a travel plan for me to go to Shanghai", the corresponding questions in the second example include "Please design a travel plan for me to go to Shanghai", "Please design a travel plan for me to go to Tibet", "Please design a travel plan for me to go to Qinghai", etc.

[0079] By constructing a second template for each topic using the first major language model, and generating multiple second examples for each second template based on the first major language model, this application can generate multiple second examples for each topic according to the first major language model. Therefore, this application can expand a small number of examples by following the generation rules of the first major language model, while ensuring the coverage of the expanded examples.

[0080] Quality inspection module 603 is used to perform quality inspection on the second example based on the first major language model.

[0081] Suppose there are T topics, and correspondingly T second templates. Each second template has M second examples, resulting in a total of T*M second examples. We then use the first language model to perform quality checks on each of these T*M second examples, extracting N valid examples, where N is less than or equal to...

[0082] The quality of the second examples was checked using three dimensions: reasonableness, format consistency, and diversity. Invalid examples were removed. Reasonableness refers to the degree to which the answer in a second example is appropriate for the question, specifically including relevance, sufficiency, and correctness. Format consistency refers to whether the format of the second example is consistent with the format of the first example entered by the user. Diversity refers to whether the differences between multiple second examples meet preset requirements, in order to remove duplicates of identical or similar second examples.

[0083] Each second example undergoes a quality check based on three dimensions: reasonableness, format consistency, and diversity. Invalid examples are removed, and the remaining second examples are used to build the example library. Invalid examples are those that do not meet the requirements of reasonableness, format consistency, and / or diversity.

[0084] The quality inspection module 603 is optional. In embodiments of this application that do not include the quality inspection module 603, the second example can be directly used to generate the example library; this scenario is suitable when the second example is of high quality.

[0085] Example library generation module 604 is used to generate an example library based on the second example.

[0086] The example library includes multiple second examples, each consisting of a question and an answer. When a user query is performed based on the second largest language model, the example library provides resources for example retrieval for the second largest language model. The retrieved second examples that match the user query are then injected into the prompt words as second examples to improve the effectiveness of the examples injected into the prompt words.

[0087] In this embodiment of the application, in order to facilitate the retrieval of examples, the example library generation module 604 also includes an index for the example library using a vector engine. Vector engines are commonly used technologies in the prior art, such as Faiss, Proxima, Vearch, etc., and are not specifically limited here.

[0088] As one implementation approach, existing vector retrieval methods within the vector engine are used to retrieve examples. Correspondingly, an example library is built using these existing methods. Specifically, the question in the second example is vectorized to obtain a vectorized representation of the question, and an example library is constructed based on the second example and the vectorized representation of the question.

[0089] As an implementation approach, to improve the accuracy of example retrieval, an expert model is introduced during the construction of the vector engine. The expert model includes an expert route and multiple experts. The expert route is used to calculate the weight of each expert. Each expert is a pre-trained retrieval model built based on the vector engine, and each expert corresponds to a dimension, including semantic relevance, domain relevance, formal relevance, topic relevance, format similarity, quality assessment, and timeliness.

[0090] In the expert model, expert routes are trained based on questions in the corpus, experts are trained based on question-answer pairs in the corpus, and expert routes are represented using vectorized question representations.

[0091] One implementation approach is to set up a machine learning platform on the client side, with an interface providing a first example import method. Users simply import the first example through this interface to generate an example library on the server side, without needing to concern themselves with the intermediate execution process. For example, in the client interface, users can input multiple first examples simultaneously. After clicking the "Generate" button, the server executes relevant code to implement the functions of the second example generation module 602 and the example library generation module 603, thereby generating the example library and storing it on the server. During the generation of the example library based on multiple user-input first examples, the user can see the generation progress on the interface and be prompted to check the results upon completion. The generation progress can be the completion status of each module or a progress bar for the entire generation process. It should be noted that the meta-template can be a meta-template selected by the platform based on the first example or a default meta-template set by the system.

[0092] As another implementation method, the operation interface of each module can be set in the platform's operation interface, or several modules can be integrated, and operation interfaces can be set for each module or the integrated module.

[0093] In this embodiment of the application, multiple examples can be expanded from the few examples provided to the user based on the meta template and the first major language model, and an example library can be built. This improves the efficiency of example generation, increases the coverage of examples, and provides rich retrieval example resources for user queries. This ensures that examples matching the user query can be retrieved, improves the effectiveness of examples injected into prompt words, and enhances the repair results of faulty use cases.

[0094] See Figure 9 This application provides a query device 900, which includes:

[0095] The second receiving module 901 is used to receive user queries.

[0096] User queries are information that represents a user's query intent. Machine learning platforms provide input interfaces for user queries.

[0097] Example retrieval module 902 is used to retrieve multiple second examples related to the user query from the example library;

[0098] The example library stores multiple second examples. Retrieving second examples from the example library yields multiple second examples relevant to the user query. The number of retrieved second examples is less than the total number of second examples stored in the example library. Since the example library is built on a vector engine, during retrieval, the user query is vectorized. The vectorized user query is then used to retrieve second examples from the example library. The number of second examples can be one or more.

[0099] Example retrieval can be performed using vector engines such as FAISS and Annoy, or it can be performed using expert models. The expert model includes an expert route and multiple experts. The expert route is used to calculate the weight of each expert. Each expert is a trained retrieval model, and each expert corresponds to a dimension, including semantic relevance, topic relevance, format similarity, quality assessment, and timeliness. During example retrieval, the weight of the user query relative to each expert is first determined based on the user query and the expert route. Then, the search is performed on each expert according to the user query to find a second example that matches the user query. Finally, the search results of each expert are comprehensively calculated based on their weights, and the second example with a score greater than a first threshold or ranked in the top N is selected as the second example. The calculation of expert weights and the retrieval using each expert are both performed using a vectorized representation of the user query. Furthermore, the first threshold and N are fixed values, where N is a positive integer.

[0100] Using expert models for example retrieval allows for the retrieval of more accurate examples from the example library, enabling timely intervention in the generation of faulty use cases. This improves the effectiveness of examples injected into the prompt words and helps enhance the repair results of faulty use cases.

[0101] Figure 10 This is an example diagram illustrating example retrieval using an expert model, divided into an offline training phase and an online usage phase. In the offline training phase, the example retrieval engine is trained using constructed example question-answer pairs. The example retrieval engine includes expert routes and P retrieval experts, where P is a positive integer. Training dimensions include semantic relevance, topic relevance, format similarity, quality assessment, and timeliness, with one expert corresponding to one dimension. In the online usage phase, the trained example retrieval engine is used to retrieve recall examples relevant to the user's query from the example library.

[0102] The prompt word generation module 903 is used to inject multiple second examples into the prompt word template to generate prompt words.

[0103] The prompt word template is a pre-set template; please refer to [link / reference]. Figure 11 The prompt templates provided in this application include elements such as role, objective, skill, example, and limitation. Figure 11 The prompt template and the types and quantities of elements contained in the prompts shown are merely examples and are not intended to limit this application. In the embodiments of this application, a second example is injected into the "example" element in the prompt template to generate a prompt.

[0104] The first adjustment module 904 is used to adjust the location distribution of the second example based on the user query.

[0105] For user queries, if there are multiple second examples, their positional distribution will affect the effectiveness of the generated answer. Different user queries have different requirements for the positional distribution of the second examples. Therefore, the positional distribution of the second examples is dynamically adjusted according to the user query to generate the optimal second examples suitable for the user query, thereby generating the optimal suggestion words. Here, the positional distribution of the second examples refers to their sorting order.

[0106] The example sorter is a pre-trained neural network model. It calculates scores for each second example based on the user query, sorts the second examples according to their scores, and filters out second examples with scores below a second threshold. This ensures the validity of second examples included in the suggestion keywords, where the second threshold is a fixed value. The scores for each second example can be calculated based on dimensions such as relevance to the user query, closeness to the user query, topic relevance, and / or popularity.

[0107] In contrast to the static distribution of examples injected into prompts, the example sorter can dynamically adjust the distribution of each example injected into prompts based on the user query to generate the optimal example distribution suitable for the user query, thereby improving the effectiveness of the generated prompts and improving the repair effect of faulty test cases.

[0108] It should be noted that for a user query, if the position distribution of the second example returned from the example library is already the optimal result, then its position distribution does not need to be adjusted.

[0109] The second adjustment module 905 is used to adjust the positional distribution of elements in the prompt words according to the user query.

[0110] For user queries, the positional distribution of elements affects the effectiveness of the generated answer, and different user queries have different requirements for element positional distribution. Therefore, after the content of each element is filled in, the element sorter adjusts the position of the elements in the prompt word template based on the user query to generate the optimal prompt words suitable for the user query, and thus generate the optimal answer. Here, element positional distribution refers to the positional distribution of each element in the prompt words.

[0111] For example in Figure 11 In this context, the positions of the elements "skills" and "examples" in the prompt differ from their positions in the prompt template. Those skilled in the art should understand that these are merely examples. In contrast to a static element sorting, the element sorter dynamically adjusts the positional distribution of each element in the prompt based on the user query, generating an optimal element positional distribution suitable for the user's query. This improves the effectiveness of the generated prompts and, consequently, enhances the repair performance of faulty test cases.

[0112] It should be noted that for a user query, if the positional distribution of elements in the suggestion words is already the optimal result, there is no need to adjust the positional distribution of elements in the suggestion words. In this case, the positional distribution of elements in the suggestion words is the same as the suggestion word template.

[0113] It should be noted that, Figure 11 The types, quantities, and location distribution of elements in the prompt template, as well as the number of examples, are merely illustrative and not intended to limit this application. Users can configure the types, quantities, content, and location distribution of elements in the prompt template through the interface provided by the machine learning platform. The number of examples can be one or multiple.

[0114] Similar to the example sorter, the feature sorter is also a trained neural network model. The example sorter and the feature sorter can use the same model or different models.

[0115] It should be noted that the first adjustment module 904 and the second adjustment module 905 are optional modules.

[0116] The answer generation module 906 is used to input the prompt words and user query into the second language model and output the answer corresponding to the user query.

[0117] The second language model can be the same as or different from the first language model.

[0118] As one implementation method, a user interface is set up in the machine learning platform. The interface provides a user query input interface. Users only need to enter their query into the interface to see the generated answer in the results display area, without having to pay attention to the intermediate execution process.

[0119] As one implementation method, the machine learning platform has an operation interface. The operation interface of each module can be set in the interface, or several modules can be integrated, and operation interfaces can be set for each module or the integrated module.

[0120] In this embodiment, examples retrieved from the example library that match the user's query are injected into the prompt words. This allows the example library to be applied to the user's access to the large model application, generating a response corresponding to the user's query. Since the examples retrieved from the example library are those that match the user's query, this not only effectively controls the number of examples injected into the prompt words but also improves the effectiveness of the examples injected, thus enhancing the repair results of faulty use cases.

[0121] It should be noted that the functionality of the example library construction apparatus provided in this application embodiment can be implemented in the offline stage, while the functionality of the query apparatus can be implemented in the online stage. The example library generated in the offline stage is used to provide a dataset for example retrieval in the online stage. The offline stage and the online stage can run on the same platform of the same client, or on different platforms of the same client, or on different platforms of different clients. The output of the online stage can also be used as the input of the offline stage to update the example library.

[0122] This application provides a large language model application system, including the example library construction device 600 and the query device 900 provided in the foregoing embodiments.

[0123] See Figure 12 This application provides a method for building an example library, including the following steps:

[0124] S1201, Receive the first example.

[0125] The first example is the initial example of user input. There are one or more first examples. Compared with the expanded second examples, the first example of user input is a small number of examples.

[0126] S1202. Input the meta-template and the first example into the first large language model to generate the second example.

[0127] The purpose of expanding the first example is to increase the number and topics of the few first examples input by the user, so as to improve the efficiency of generating the second example and increase the coverage of the second example.

[0128] See Figure 13 S1202 includes the following sub-steps:

[0129] S12021. Determine the topic related to the first example;

[0130] There can be one or more topics associated with a first example. In this embodiment, the determined topics associated with a first example are the sum of all topics associated with the first examples, and the number can be one or more.

[0131] S12022. Construct a first template based on the topic, meta template, and first example related to the topic, and input the first template into the first large language model to obtain the second template.

[0132] The number of first examples related to a topic can be one or more. A first template can contain the original content of the first examples under the corresponding topic, or it can contain modified first example content. A first template is built for a topic, and a second template is generated from a first template.

[0133] S12023. Construct a third template based on the second template, and input the third template into the first large language model to obtain the second example.

[0134] The first major language model performs quality checks on each second example according to three dimensions: reasonableness, format consistency, and diversity, in order to delete invalid examples and use the remaining second examples to build an example library.

[0135] S1203. Perform quality checks on the second example based on the first major language model.

[0136] The first language model performs quality checks on each second example according to three dimensions: reasonableness, format consistency, and diversity. Invalid examples are deleted, and the remaining extended examples are used to generate the example library.

[0137] It should be noted that step S1203 is optional. If the quality of the second example is better, step S1203 does not need to be performed to improve processing efficiency.

[0138] S1204. Generate a sample library based on the second example.

[0139] When a user query is performed based on the second largest language model, the example library provides resources for example retrieval for the second largest language model. The second example that matches the user query is used as the second example and injected into the prompt words to improve the effectiveness of the examples injected into the prompt words.

[0140] As one implementation approach, the example library is constructed using existing vector engines. Specifically, the problem in the second example is vectorized to obtain a vectorized representation of the problem, and the example library is generated based on the second example and the vectorized representation of the problem.

[0141] As an implementation approach, an expert model is introduced during the construction of the vector engine to improve the accuracy of example retrieval.

[0142] In this embodiment of the application, multiple examples can be expanded from the few examples provided to the user based on the meta template and the first major language model, and an example library can be built. This improves the efficiency of example generation, increases the coverage of examples, and provides rich retrieval example resources for user queries, so as to ensure that examples matching the user query can be retrieved. This improves the quality of examples injected into the prompt words and enhances the repair results of faulty use cases.

[0143] See Figure 14 This application provides a query method, including the following steps:

[0144] S1401, Receive user queries.

[0145] S1402. Retrieve multiple second examples related to the user query from the example library.

[0146] Example retrieval is performed using an expert model. The expert model includes an expert route and multiple experts. The expert route is used to calculate the weight of each expert. Each expert is a trained retrieval model, and each expert corresponds to a dimension, including semantic relevance, domain relevance, formal relevance, topic relevance, format similarity, quality assessment, and timeliness. During example retrieval, firstly, the weight of the user query relative to each expert is determined based on the user query and the expert route. Then, the expert is searched according to the user query to find a second example matching the user query. Finally, the search results of each expert are comprehensively calculated based on their weights, and the second example with a score greater than a first threshold or ranked in the top N is selected as the second example. The calculation of expert weights and the retrieval using each expert are both performed using a vectorized representation of the user query. Furthermore, the first threshold and N are fixed values, where N is a positive integer.

[0147] Using expert models for example retrieval allows for the retrieval of more accurate examples from the example library, enabling timely intervention in the generation of faulty use cases. This improves the effectiveness of examples injected into the prompt words and helps enhance the repair results of faulty use cases.

[0148] S1403. Inject multiple second examples into the prompt word template to generate prompt words.

[0149] S1404. Adjust the positional distribution of multiple second examples in the prompt words based on the user query.

[0150] For user queries, if there are multiple second examples, their positional distribution will affect the effectiveness of the generated answer. Different user queries have different requirements for the positional distribution of the second examples. Therefore, the positional distribution of the second examples should be dynamically adjusted according to the user query to generate the optimal second examples suitable for the user query, thereby generating the optimal suggestion words.

[0151] S1405. Adjust the positional distribution of multiple elements in the prompts based on user queries.

[0152] For user queries, the location distribution of elements affects the effectiveness of the generated answer, and different user queries have different requirements for element location distribution. Therefore, after the content of each element is filled in, the element sorter adjusts the position of the elements in the prompts based on the user query to generate the optimal prompts suitable for the user query, and thus generate the optimal answer.

[0153] It should be noted that steps S1404 and S1405 are optional.

[0154] S1406. Input the prompt words and user query into the second language model and output the answer corresponding to the user query.

[0155] In this embodiment, examples retrieved from the example library that match the user's query are injected into the prompt words to generate a response corresponding to the user's query. Since the examples retrieved from the example library are those that match the user's query, this not only effectively controls the number of examples injected into the prompt words but also improves the effectiveness of the injected examples, thus enhancing the repair results of faulty use cases.

[0156] It should be noted that the steps and flow in the method embodiments provided in this application correspond to the functional modules in the device embodiments.

[0157] Please see Figure 15 This is an example diagram of an embodiment of the computing device in this application.

[0158] The computing device provided in this embodiment can be a processor, a server, or a dedicated data processing device, etc. The specific form of the device is not limited in this embodiment.

[0159] The computing device 1500 may vary considerably due to different configurations or performance, and may include one or more processors 1501 and memory 1502, in which programs or data are stored.

[0160] The memory 1502 can be volatile or non-volatile memory. Optionally, the processor 1501 is one or more central processing units (CPUs), graphics processing units (GPUs), or other dedicated processors, such as Ascend. The CPU can be a single-core CPU or a multi-core CPU. The processor 1501 can communicate with the memory 1502 and execute a series of instructions stored in the memory 1502 on the computing device 1500.

[0161] The computing device 1500 also includes one or more wired or wireless network interfaces 1503, such as Ethernet interfaces.

[0162] Optionally, although Figure 15 As not shown in the diagram, the computing device 1500 may also include one or more power supplies; one or more input / output interfaces, which can be used to connect to a monitor, mouse, keyboard, touch screen device or sensing device, etc. The input / output interfaces are optional components and may or may not be present, and are not limited here.

[0163] In this embodiment, the memory 1502 in the computing device 1500 stores a computer program. The processor 1501 in the electronic device 1500 executes the computer program stored in the memory 1502, and can perform the steps of the above method embodiment. The above embodiments can be implemented entirely or partially by software, hardware, firmware, or any combination thereof. When implemented using software, they can be implemented entirely or partially in the form of a computer program product.

[0164] A computer program product includes one or more computer instructions. When the computer program product runs on a processor, the computer loads and executes the computer execution instructions, producing all or part of the processes or functions of the embodiments of this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0165] Computer-readable storage media can be any usable medium that a computer can store, or a data storage device such as a server or data center that integrates one or more usable media. The usable medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)). The computer-readable storage medium stores a computer program that, when executed on a processor, produces all or part of the processes or functions described in the embodiments of this application.

[0166] The technical solutions provided in this application have been described in detail above. Specific examples have been used in this application to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for constructing an example library, characterized in that, include: Receive the first example; The meta-template and the first example are input into the first large language model to generate the second example; Both the first example and the second example include questions and answers; the second example has more questions than the first example. A sample library is generated based on the second example; the sample library is used to provide injection examples for the second major language model, the injection examples being second examples related to user queries, and the second major language model being used to provide answers to the user queries.

2. The method as described in claim 1, characterized in that, The step of inputting the meta-template and the first example into the first large language model to generate the second example includes: Identify the topics relevant to the first example; A first template is constructed based on the topic, meta template, and the first example related to the topic, and the first template is input into the first large language model to obtain a second template; A third template is constructed based on the second template, and the third template is input into the first large language model to obtain the second example.

3. The method as described in claim 1 or 2, characterized in that, Also includes: The second example is quality checked based on the first large language model.

4. The method according to any one of claims 1-3, characterized in that, Also includes: Based on the expert model, a vector engine for the example library is constructed.

5. A query method, characterized in that, include: Receive user queries; Retrieve multiple second examples related to the user's query from the example library; The multiple second examples are injected into the prompt word template to generate prompt words; The prompt words and the user query are input into the second language model, which outputs the answer corresponding to the user query.

6. The method as described in claim 5, characterized in that, Before inputting the prompt words and the user query into the second large language model, the method further includes: Adjust the positional distribution of the multiple second examples in the prompt words based on the user query.

7. The method as described in claim 5 or 6, characterized in that, Before inputting the prompt words and the user query into the second large language model, the method further includes: Adjust the positional distribution of the multiple elements in the prompt words based on the user query.

8. The method according to any one of claims 5-7, characterized in that, The elements include role, objective, skills, example, and limitations.

9. The method according to any one of claims 5-8, characterized in that, The step of retrieving multiple second examples related to the user query from the example library includes: The user query is vectorized; The weights of multiple experts are determined based on the user query with a vectorized representation; The multiple experts search the example library for a second example that matches the user's query; The score of the second example that matches the user query is calculated based on the weights. Based on the score, a second example is selected from the second examples that match the user query.

10. An apparatus for building an example library, characterized in that, include: The first receiving module is used to receive the first example; The second example generation module is used to input the meta-template and the first example into the first large language model to generate a second example; both the first example and the second example include questions and answers; the number of second examples is greater than that of the first examples; An example library generation module is used to generate an example library based on the second example; the example library is used to provide a second example for the large language model; the example library is used to provide an injection example for the second large language model, the injection example being a second example related to the user query, and the second large language model is used to provide an answer to the user query.

11. The apparatus as claimed in claim 10, characterized in that, The second example generation module specifically includes: The topic determination module is used to determine the topic related to the first example; The second template generation module is used to construct a first template based on the topic, meta template and the first example related to the topic, and input the first template into the first large language model to obtain the second template; The second example generation module is used to construct a third template based on the second template and input the third template into the first large language model to obtain the second example.

12. The apparatus as claimed in claim 10 or 11, characterized in that, It also includes a quality inspection module; The quality inspection module is used to perform quality inspection on the second example based on the first large language model.

13. The apparatus according to any one of claims 10-12, characterized in that: The example library generation module is also used to construct a vector engine for the example library based on the expert model.

14. A query device, characterized in that, include: The second receiving module is used to receive user queries; The example retrieval module is used to retrieve multiple second examples related to the user's query from the example library; The prompt word generation module is used to inject the plurality of second examples into the prompt word template to generate prompt words; The answer generation module is used to input the prompt words and the user query into the second language model and output the answer corresponding to the user query.

15. The apparatus as claimed in claim 14, characterized in that, The device also includes a first adjustment module; The first adjustment module is used to adjust the location distribution of the multiple second examples according to the user query.

16. The apparatus as claimed in claim 14 or 15, characterized in that, The prompt word includes multiple elements; the device also includes a second adjustment module; The second adjustment module is used to adjust the positional distribution of the multiple elements in the prompt words according to the user query.

17. The apparatus according to any one of claims 14-16, characterized in that: The elements include role, objective, skills, example, and limitations.

18. The apparatus according to any one of claims 14-17, characterized in that, The example retrieval module specifically includes: The user query is vectorized; The weights of multiple experts are determined based on the user query with a vectorized representation; The multiple experts search the example library for a second example that matches the user's query; The score of the second example that matches the user query is calculated based on the weights. Based on the score, a second example is selected from the second examples that match the user query.

19. A large language model application system, comprising: The first receiving module is used to receive the first example; The second example generation module is used to input the meta-template and the first example into the first large language model to generate a second example; both the first example and the second example include questions and answers; the number of second examples is greater than that of the first examples; An example library generation module is used to generate an example library based on the second example; the example library is used to provide a second example for the large language model. The second receiving module is used to receive user queries; The example retrieval module is used to retrieve multiple second examples related to the user's query from the example library; The prompt word generation module is used to inject the plurality of second examples into the prompt word template to generate prompt words; The answer generation module is used to input the prompt words and the user query into the second language model and output the answer corresponding to the user query.

20. An electronic device, characterized in that, The device includes a processor and a memory interconnected thereto, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is used to invoke the program instructions to execute the method as described in any one of claims 1-9.

21. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-9.

22. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-9.

Images