Prompt text generation method and device based on large language model and storage medium

Abstract

The invention provides a prompt text generation method and device based on a large language model and a storage medium. The invention relates to the technical field of artificial intelligence, in particular to the field of intelligent search, large model and prompt text optimization, and can be applied to application scenes such as an intelligent customer service system, an automatic question and answer system, a content generation tool, a code auxiliary tool, a data analysis assistant, an education tutoring system and a medical diagnosis auxiliary system. According to the specific implementation scheme, output content generated by a first large language model for a plurality of first Query based on a first prompt text is obtained; obtaining an evaluation score and explanation of the output content generated by the second large language model based on the expected response behavior; determining a second Query from the plurality of first Query, based on the expected response behavior, the second Query and the corresponding output content and explanation, generating a modification suggestion for the first prompt text; and modifying the first prompt text based on the modification suggestion to obtain a second prompt text.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the fields of intelligent search, large models, and prompt text optimization. It can be used in application scenarios such as intelligent customer service systems, automatic question-and-answer systems, content generation tools, code assistance tools, data analysis assistants, educational counseling systems, and medical diagnosis assistance systems. It specifically relates to a prompt text generation method, device, and storage medium based on a large language model. Background Art

[0002] With the advancement of machine learning, researchers have tried to automatically optimize Prompt. For example, some people have proposed a gradient-based search method to optimize Prompt, but such methods still require a lot of computing resources and data, and are difficult to quickly and efficiently adapt to needs in complex and diverse scenarios, especially in scenarios such as intelligent customer service and automatic question and answer that process a large amount of unstructured input. Summary of the invention

[0003] The present invention provides a method, device and storage medium for generating prompt text based on a large language model.

[0004] According to one aspect of the present disclosure, a method for generating prompt text based on a large language model is provided, the method comprising:

[0005] Obtain output content generated by the first language model for a plurality of first queries based on the first prompt text;

[0006] Obtain the evaluation score and explanation of the output content generated by the second largest language model based on the expected response behavior;

[0007] Determine a second query from the plurality of first queries, the second query being a first query whose evaluation score is lower than a first threshold;

[0008] Generate modification suggestions for the first prompt text based on the expected response behavior, the second Query and its corresponding output content and explanation;

[0009] The first prompt text is modified based on the modification suggestion to obtain a second prompt text.

[0010] According to another aspect of the present disclosure, a device for generating prompt text based on a large language model is provided, the device comprising:

[0011] A first acquisition unit, configured to acquire output content generated by a first language model for a plurality of first queries based on a first prompt text;

[0012] A second acquisition unit is used to obtain an evaluation score and an explanation of the output content generated by the second language model based on the expected response behavior;

[0013] A first determining unit, configured to determine a second query from a plurality of first queries, the second query being a first query having an evaluation score lower than a first threshold;

[0014] A first generating unit, configured to generate a modification suggestion for the first prompt text based on the expected response behavior, the second Query and its corresponding output content and explanation;

[0015] The second generating unit is used to modify the first prompt text based on the modification suggestion to obtain a second prompt text.

[0016] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0020] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0021] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0022] By adopting the solution disclosed in the present invention, Prompt can be automatically optimized in a systematic and low-cost manner, and can adapt to the needs of different business scenarios.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0025] Figure 1 is a flowchart of a method for generating prompt text based on a large language model according to an embodiment of the present disclosure;

[0026] Figure 2is a schematic diagram of the architecture of a prompt text generation system based on a large language model according to an embodiment of the present disclosure;

[0027] Figure 3 is a flow chart of optimizing prompt text based on a large language model according to an embodiment of the present disclosure;

[0028] Figure 4 is a structural schematic diagram of a prompt text generation device based on a large language model according to an embodiment of the present disclosure;

[0029] Figure 5 is a schematic diagram of a scenario of a method for generating prompt text based on a large language model according to an embodiment of the present disclosure;

[0030] Figure 6 It is a structural diagram of an electronic device used to implement the prompt text generation method based on a large language model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0032] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them. They do not mean to limit the order or to limit them to only two. For example, the first feature and the second feature refer to two types / two features. The first feature can be one or more, and the second feature can also be one or more.

[0033] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0034] Before introducing the technical solutions of the embodiments of the present disclosure, the technical terms that may be used in the present disclosure are further explained:

[0035] Large language model: refers to a model with large-scale parameters and training data that can generate coherent and natural language.

[0036] Query: Usually a text or instruction input to a large language model to guide the model to generate the corresponding output content. Users submit queries to request the model to generate corresponding answers, text generation, translation, summary and other tasks. Queries can be simple sentences, paragraphs, or even more complex instructions, depending on the type of task the user wants the model to complete.

[0037] Expected response behavior (Expect): refers to the standard or pattern that the output content generated by the large language model should meet.

[0038] Suggestion: A specific suggestion for improving the prompt text.

[0039] In the related technologies, the prompt optimization technology has the following main disadvantages:

[0040] 1. High labor cost: Traditional manual trial and error methods require a lot of manpower input. Especially in complex scenarios, the optimization process may take days or even weeks, greatly increasing project costs.

[0041] 2. Lack of systematicity: Existing methods often lack a systematic optimization process, resulting in unstable optimization results that are difficult to replicate and extend to other scenarios.

[0042] 3. Poor adaptability: Templated and rule-based methods are difficult to adapt to novel or complex business scenarios, limiting the application scope of large language models.

[0043] 4. High evaluation cost: The evaluation method based on the Golden Answer requires labeling the standard answer for each query, which is unrealistic in large-scale applications and cannot meet the needs of rapid iteration.

[0044] 5. Unstable optimization effect: Although machine learning optimization methods have certain adaptability, they often require a large amount of training data and tuning, and the optimization effect is unstable and difficult to ensure consistency.

[0045] 6. Difficulty in handling complex tasks: Existing methods perform poorly when handling tasks that require multi-step reasoning or complex logic, and it is difficult to effectively optimize Prompt to improve model performance.

[0046] 7. Lack of feedback loop: Most existing methods lack an effective feedback mechanism, making it difficult to dynamically adjust optimization strategies based on model output and actual results.

[0047] The above shortcomings seriously limit the effectiveness and efficiency of large language models in practical applications. There is an urgent need for an automated, systematic, low-cost and highly adaptable (universal) Prompt optimization solution to solve these problems.

[0048] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, the present disclosure proposes a prompt text generation method based on a large language model, which can realize automatic optimization of Prompt through a universal scorer, a suggestion generator and an optimizer, significantly reduce the manpower cost in the Prompt optimization link, realize an efficient and low-cost Prompt optimization process, and can adapt to the needs of different business scenarios.

[0049] The present disclosure provides a method for generating prompt text based on a large language model. Figure 1 It is a flow chart of a prompt text generation method based on a large language model according to an embodiment of the present disclosure. The prompt text generation method based on a large language model can be applied to a prompt text generation device based on a large language model. The prompt text generation device based on a large language model is located in an electronic device. The electronic device includes but is not limited to fixed devices and / or mobile devices. For example, fixed devices include but are not limited to servers, and the server can be a cloud server or an ordinary server. For example, mobile devices include but are not limited to: mobile phones, tablet computers. In some possible implementations, the prompt text generation method based on a large language model can also be implemented by a processor calling computer-readable instructions stored in a memory. For example Figure 1 As shown, the prompt text generation method based on the large language model includes:

[0050] S101, obtaining output content generated by a first language model for a plurality of first queries based on a first prompt text;

[0051] S102, obtaining an evaluation score and explanation of the output content generated by the second largest language model based on the expected response behavior;

[0052] S103, determining a second query from the plurality of first queries, where the second query is a first query whose evaluation score is lower than the first threshold;

[0053] S104, generating a modification suggestion for the first prompt text based on the expected response behavior, the second Query and its corresponding output content and explanation;

[0054] S105: Modify the first prompt text based on the modification suggestion to obtain a second prompt text.

[0055] In the disclosed embodiment, the first prompt text is a prompt text for multiple first queries at a first moment, and the second prompt text is a prompt text for multiple first queries at a second moment, and the second moment is later than the first moment.

[0056] In the disclosed embodiment, the second prompt text is a revised version based on the modification suggestions for the first prompt text. Compared with the first prompt text, the second prompt text shows better performance in terms of applicability.

[0057] In the disclosed embodiment, the first large language model and the second large language model may be different large language models or the same large language model, that is, a large language model is used to generate results based on the query output and to output scores and explanations for the generated results based on the expected response behavior. The first language model and the second language model include, but are not limited to: a pre-trained language generation model, a language model based on a transformer architecture, or a language generation model optimized for a specific domain.

[0058] In the disclosed embodiment, the expected response behavior is to evaluate whether the output content conforms to a certain specific, expected response mode or standard.

[0059] In the disclosed embodiment, the evaluation score is determined by the second largest language model in combination with the expected reply behavior and according to a preset scoring standard, and the preset scoring standard is adapted to the business scenario to which the Query belongs.

[0060] In some embodiments, obtaining output content generated by the first large language model for multiple first queries based on the first prompt text includes: determining an initial prompt text, which is a starting point for the large language model to generate corresponding content based on it; using the first large language model, according to the first prompt text, generating output content for multiple different first queries (these queries can be questions, instructions or topics related to the first prompt text), and these output contents will serve as the basis for subsequent evaluation.

[0061] In some embodiments, obtaining an evaluation score and explanation of the output content generated by the second largest language model based on the expected response behavior includes: using the second largest language model (which can be another trained large language model, or a language model specifically used for evaluation) to evaluate the output content generated by the first largest language model, and the second largest language model generates an evaluation score for each output content, as well as an explanation of why this score is given.

[0062] In some embodiments, determining a second query from multiple first queries includes: based on the evaluation scores given by the second largest language model, screening out first queries whose evaluation scores are lower than a first threshold, the output content corresponding to these queries may not meet the expected response behavior, or the quality is low, and marking these screened queries as second queries as a reference for subsequent modification of the prompt text.

[0063] In some embodiments, based on the expected response behavior, the second Query and its corresponding output content and explanation, generating modification suggestions for the first prompt text includes: analyzing the explanation given by the second largest language model to understand why the output content corresponding to the second Query does not meet the expected response behavior. Based on these analyses, generating modification suggestions for the first prompt text. These suggestions are intended to improve the prompt text so that it can guide the first largest language model to generate output content that is more consistent with the expected response behavior.

[0064] In some embodiments, the first prompt text is modified based on the modification suggestion to obtain the second prompt text, including: modifying the first prompt text according to the generated modification suggestion; the modified text is the second prompt text, which should be able to guide the first language model to generate output content with higher quality and more in line with the expected response behavior.

[0065] For example, suppose the first prompt text is "Please write an article about the development of artificial intelligence". The first language model generates output content for multiple queries (such as "the history of artificial intelligence", "the current situation of artificial intelligence", "the future of artificial intelligence", etc.) based on this prompt. Then, the second language model evaluates these output contents according to the expected response behavior (such as the clear structure of the article, accurate content, fluent language, etc.), and gives evaluation scores and explanations. Suppose that the output content evaluation score corresponding to one of the queries "the future of artificial intelligence" is low because the content is too abstract and lacks specific examples. Based on this evaluation result, a modification suggestion for the first prompt text can be generated, such as "add requirements for specific content and examples of the article in the prompt text". Finally, the first prompt text is modified according to this modification suggestion to obtain the second prompt text, such as "Please write an article about the development of artificial intelligence, which is required to include specific historical events, current application examples, and prospects for the future". In this way, the prompt text can be continuously optimized to improve the quality and accuracy of the text generated by the large language model.

[0066] According to the technical solution of the embodiment of the present disclosure, the second largest language model evaluates the output content of the first largest language model based on the expected response behavior to obtain an evaluation score and an explanation; based on the expected response behavior, the second query with an evaluation score lower than the first threshold and its corresponding output content and explanation, a modification suggestion for the first prompt text is generated; based on the modification suggestion, the first prompt text is modified to obtain the second prompt text; in this way, the prompt can be optimized automatically, systematically, at low cost and with high adaptability, thereby improving the quality and efficiency of products based on the large language model. Since there is no need to generate a standard answer for each query, the expected response behavior is used instead of the standard answer, which not only greatly reduces the data annotation cost, but also significantly reduces the manpower investment in the prompt tuning link and reduces the evaluation cost. The systematic optimization process can complete the tuning of a large number of prompts in a short time and accelerate product iteration; through continuous optimization, the continuous improvement of model performance is achieved, the quality, relevance and consistency of the model output are significantly improved, and it can quickly adapt to different business scenarios and improve the application flexibility of the product in various fields. Since the optimization process is transparent and traceable, it helps to understand and further improve the model behavior and provide the interpretability of the optimization.

[0067] In some embodiments, obtaining an evaluation score and explanation of the output content generated by the second largest language model based on the expected response behavior includes: determining the business scenario to which the multiple first queries belong; determining a scoring standard that matches the business scenario; generating a scoring score for the output content for each first Query according to the scoring standard; and generating an explanation of the output content for each first Query based on the expected response behavior.

[0068] In the disclosed embodiment, the second largest language model is a model used to evaluate the output content, which may not be the most advanced or complex model, but performs well in specific tasks or scenarios.

[0069] In the embodiments of the present disclosure, the expected response behavior is the response standard that the user expects to obtain from the system or model in a given business scenario.

[0070] In some implementations, the business scenario to which each first query (ie, user input or request) belongs is clarified. Here, the business scenario includes but is not limited to customer service consultation, product recommendation, technical support, question answering, and the like.

[0071] In some implementations, a scoring standard that matches the determined business scenario is selected. Here, the scoring standard may include multiple dimensions such as accuracy, relevance, timeliness, and friendliness. Different business scenarios may require different scoring standard combinations or weight allocations.

[0072] In some implementations, for each first query, the second largest language model will score its output content according to the selected scoring criteria. This score reflects the performance of the output content in multiple dimensions, such as whether it accurately answers the user's question, whether it is highly relevant to the user's needs, whether it is timely, etc.

[0073] In some embodiments, in addition to the scoring score, the second language model also generates explanations for each query based on expected response behavior. These explanations provide the reasoning behind the score, helping users or system developers understand why the output content received such a score. For example, if the output content scores low in accuracy, the explanation may point out errors or omissions.

[0074] For example, suppose there is a customer service system of an e-commerce platform, and a user enters a query about product returns and exchanges. The second largest language model first determines that this query belongs to the "customer service consultation" business scenario, and then matches it with relevant scoring criteria, such as "accuracy", "friendliness" and "timeliness". Then, the model scores the system's response based on these criteria and generates an explanation to explain why this response scored high in "accuracy" but needs to be improved in "friendliness". This makes it easier for system developers to optimize the response content based on these feedbacks.

[0075] In this way, by matching business scenarios and scoring criteria, the second largest language model can more accurately evaluate the quality of the output content and improve the accuracy of the evaluation; providing explanations not only increases the transparency of the evaluation process, but also helps system developers understand how to improve the output content. This method can be integrated into the automation system to achieve real-time evaluation and feedback on the output content.

[0076] In some embodiments, multiple first queries correspond to the same expected response behavior, and the expected response behavior is a response standard that is suitable for the business scenario.

[0077] In the disclosed embodiment, the first Query is a request or question raised by a user or a system to the first language model.

[0078] In the disclosed embodiments, the business scenario is the specific environment and background in which a user or system uses a model or service, which determines the specific content and requirements of the expected response behavior.

[0079] In the disclosed embodiment, although multiple first queries may involve different specific topics or details, they all correspond to the same expected response behavior. This expected response behavior is determined according to the business scenario, and it represents the response standard that the user or system expects in the scenario.

[0080] In some implementations, the specific type and characteristics of the business scenario are clarified, such as product consultation on an e-commerce platform, financial service consultation on a bank, or health consultation on a medical institution. According to the characteristics of the business scenario, the expected response behavior corresponding to it is defined. This behavior may include multiple aspects such as timeliness, accuracy, professionalism, and friendliness of the response.

[0081] In some implementations, multiple first queries are designed based on business scenarios and expected response behaviors. These queries cover various questions or requests that users may raise, while ensuring that they are all related to the expected response behaviors.

[0082] In some embodiments, the first language model is used to generate output content for each first query according to the first prompt text. The second language model is used to evaluate the generated output content according to the expected response behavior (here specifically referring to the response standard that is suitable for the business scenario), and a score and explanation are given. According to the evaluation results of the second language model, it is analyzed which query output content meets the expected response behavior, which does not meet the expected response behavior, and what are the reasons for the non-compliance. Based on the evaluation results and analysis, the first prompt text is modified and optimized to improve the quality of the output content generated by the first language model so that it is more in line with the expected response behavior.

[0083] For example, suppose that in the customer service system of an online education platform, the business scenario is "course consultation". In this scenario, the expected response behavior may include "responding to user questions in a timely manner", "providing accurate course information", "answering user questions in a friendly manner", etc. Based on this business scenario and expected response behavior, multiple first queries can be designed, such as "What is the class time for this course?", "Who is the teacher of this course?", "Which group of people is suitable for this course?", etc. Then, the first language model is used to generate output content for each query, and the second language model is used to evaluate the output content based on the expected response behavior. If it is found that the output content of a query has a low score in "providing accurate course information", the first prompt text can be modified and optimized to improve the quality of the generated output content.

[0084] In this way, by clarifying the expected response behavior and adapting it to the business scenario, it is possible to ensure that the generated output content is more in line with the expectations of the user or system, thereby improving the response quality. Multiple first queries correspond to the same expected response behavior, which helps to ensure that the output content generated on different topics or details remains consistent and coherent overall. High-quality responses and consistent output content can improve user satisfaction with the system or service, thereby optimizing the user experience.

[0085] In some embodiments, based on the expected response behavior, the second Query and its corresponding output content and explanation, modification suggestions for the first prompt text are generated, including: in the case of multiple second Queries, for each second Query corresponding to the output content and explanation, combined with the expected response behavior, modification suggestions for the first prompt text are generated for each second Query.

[0086] In the embodiment of the present disclosure, the second query is the first query whose evaluation score is lower than the first threshold, that is, the query whose generated output content does not meet the expected response behavior or has low quality.

[0087] In the disclosed embodiment, the modification suggestion is a specific suggestion for improving the first prompt text based on the output content and explanation and the expected response behavior.

[0088] In some embodiments, when there are multiple second queries, the output content and explanation of each second query are analyzed; for each second query, its corresponding output content and the explanation given by the second largest language model are carefully analyzed; the aspects in which the output content does not conform to the expected response behavior, as well as the specific problems or defects mentioned in the explanation, are understood, and modification suggestions for the first prompt text are generated.

[0089] In some embodiments, the output content and explanation of each second Query are compared with the expected response behavior. It is determined which deficiencies lead to the output content not meeting expectations, such as accuracy, completeness, relevance of information, or clarity of expression. Based on the above analysis, modification suggestions for the first prompt text are generated for each second Query. These suggestions should be aimed at improving the prompt text so that it can guide the first language model to generate output content that is more in line with the expected response behavior.

[0090] In some embodiments, modification suggestions may include adding specific information, adjusting language style, clarifying instruction requirements, etc. For example, if the output content has problems with accuracy, it may be suggested to add more specific details or restrictions to the prompt text; if the output content lacks clarity of expression, it may be suggested to adjust the language structure or add explanatory sentences.

[0091] In some implementations, after generating modification suggestions for the first prompt text for each second query, these modification suggestions are comprehensively evaluated to ensure that the modification suggestions not only solve the current problem but also do not introduce new deficiencies or contradictions. At the same time, the feasibility and implementation cost of the modification suggestions are considered.

[0092] In some implementations, based on the results of a comprehensive evaluation of the modification suggestions, the modification suggestions are optimized and iterated to ensure that the final modification suggestions are both effective and practical.

[0093] For example, suppose that in a travel planning service, the business scenario is "making a travel plan". The expected response behavior may include "providing a detailed itinerary", "including necessary transportation and accommodation information", "ensuring the feasibility and safety of the plan", etc. Among multiple first queries, two second queries received low evaluation scores. The first second query is "How to arrange the travel plan from Shanghai to Beijing", and its output content is insufficient in "providing a detailed itinerary"; the second second query is "I need a travel plan suitable for family travel", and its output content is insufficient in "including necessary transportation and accommodation information". For these two second queries, the following modification suggestions can be generated: For the first second query, it is recommended to add "Please provide a detailed itinerary from the departure point to the destination, including the visiting time, transportation method and route of each attraction" in the first prompt text; For the second second query, it is recommended to add "Please make a travel plan for family travel including necessary transportation and accommodation information, and ensure the comfort and safety of the plan" in the first prompt text; By implementing these modification suggestions, the first language model is guided to better meet the expectations and needs of users when generating output content.

[0094] In this way, by generating specific modification suggestions for each second query, the first prompt text can be improved more effectively, thereby improving the quality of the output content generated by the first language model. Each modification suggestion is generated based on the specific output content and explanation as well as the expected response behavior, so it is highly targeted and practical. The prompt text is continuously analyzed and improved based on the generated modification suggestions, thereby promoting the continuous optimization and progress of the first language model in generating output content.

[0095] In some embodiments, based on the expected response behavior, the second query and its corresponding output content and explanation, a modification suggestion for the first prompt text is generated, including: when there are multiple second queries, the multiple second queries are classified according to business characteristics to form multiple batches; wherein the number of second queries contained in each batch is less than the total number of second queries; for each batch, the output content and explanation corresponding to the second queries in the same batch are summarized, and combined with the expected response behavior, a modification suggestion for the first prompt text corresponding to the batch is jointly generated.

[0096] In some implementations, first, multiple second queries are classified according to business characteristics or similarity of query content. The basis for classification may be the subject, type, priority, etc. of the query. Then, the classified second queries are divided into multiple batches, and the number of second queries contained in each batch should be less than the total number of second queries, so as to ensure that the processing of each batch can be more detailed and efficient.

[0097] In some implementations, for each batch, the output content and explanations corresponding to all second queries in the batch are collected. The output content is usually a direct answer or processing result to the query, while the explanation is a detailed description or basis of the output content. These output content and explanations are summarized to form an overall understanding and evaluation of the batch of queries.

[0098] In some embodiments, based on the summarized output content and explanations, as well as the expected response behavior (i.e., how the expected system or human responds to these queries), the shortcomings of the current first prompt text in guiding queries, providing information, or solving problems are analyzed. Based on these analyses, modification suggestions for the first prompt text corresponding to the batch are jointly generated. The modification suggestions may include adjusting the wording of the prompt text, adding or deleting certain information, changing the order or structure of the prompts, etc.

[0099] Suppose an e-commerce platform has a query system where users can search for product information by entering queries. The queries entered by users are classified according to product categories (such as electronic products, clothing, home furnishings, etc.), and then each category is processed as a batch. For queries in the electronic product category, all query results and explanations about electronic products are collected, such as price, function, evaluation, etc. If it is found that users are often confused about price ranges and functional features when searching for electronic products, then guidance information about price ranges and functional features can be added to the first prompt text to better meet the user's query needs. Through such an implementation method, the performance of the query system and the user experience can be continuously optimized.

[0100] In this way, by classifying multiple second queries and processing them in batches, a large number of queries can be processed more orderly and efficiently, avoiding confusion and omissions in the processing process. By summarizing the output content and explanations of the second queries in the same batch and analyzing them in combination with the expected response behavior, problems in the first prompt text can be more accurately identified and targeted modification suggestions can be generated. The optimized first prompt text can guide user queries more accurately, provide more relevant information or solutions, and thus improve user satisfaction.

[0101] In some embodiments, the first prompt text is modified based on the modification suggestion to obtain the second prompt text, including: when there are multiple modification suggestions, the first prompt text is modified separately for each modification suggestion to obtain multiple candidate second prompt texts; and the second prompt text is determined from the multiple candidate second prompt texts.

[0102] In the disclosed embodiment, the candidate second prompt texts are multiple candidate texts obtained by modifying the first prompt text according to the modification suggestion.

[0103] In some embodiments, after obtaining modification suggestions for the first prompt text, in order to ensure the validity and accuracy of the modification, when there are multiple modification suggestions, each suggestion is analyzed one by one and applied to the first prompt text respectively. This means that for each modification suggestion, a modified candidate second prompt text will be obtained. When applying the modification suggestions, it is necessary to ensure that the modified text is fluent and reasonable in grammar, logic and context. Through the above steps, multiple candidate second prompt texts based on different modification suggestions will be obtained. These texts may be different in content, but they are all intended to optimize the first prompt text to better guide user inquiries or requests.

[0104] In some embodiments, after obtaining multiple candidate second prompt texts, it is necessary to further evaluate their effectiveness. This can be done through user testing, expert review, or data analysis. The evaluation criteria can include clarity, guidance, user satisfaction, etc. of the text. Based on the evaluation results, the best performing candidate text is selected as the final second prompt text.

[0105] For example, suppose there is a prompt text next to the search box of an online shopping platform to guide users to enter queries. The original first prompt text is "Please enter the name of the product you want to buy". Modification suggestions include: Suggestion 1: Add a search range prompt, such as "Enter the product name, brand or keyword in the search box". Suggestion 2: Emphasize the convenience of search, such as "Quickly search for the product you want, enter and search". Suggestion 3: Add promotional information, such as "Search for products and enjoy limited-time discounts". Candidate second prompt texts include: Candidate text 1: "Enter the product name, brand or keyword in the search box". Candidate text 2: "Quickly search for the product you want, enter and search". Candidate text 3: "Search for products and enjoy limited-time discounts". Through testing and data analysis, it is found that candidate text 1 performs best in guiding users to clearly enter content, so it is selected as the final second prompt text. Through such an implementation method, it can be ensured that the second prompt text not only meets business needs but also effectively improves user experience.

[0106] In this way, by applying each modification suggestion separately, multiple candidate texts can be obtained, so that the best modification plan can be selected to improve the accuracy of the modification. After generating multiple candidate texts, you can flexibly select them based on user feedback or business needs, so that the second prompt text is more suitable for different scenarios and user needs. The optimized second prompt text can more accurately guide user queries or requests, thereby improving the user experience.

[0107] In some embodiments, the first prompt text is modified based on the modification suggestion to obtain the second prompt text, including: when there are multiple modification suggestions, the specific contents of the multiple modification suggestions are integrated to modify the first prompt text to obtain the second prompt text.

[0108] In some implementations, the intention and specific changes behind each modification suggestion are understood; similar, complementary or repeated suggestions are classified and integrated to avoid redundant or contradictory information in the second prompt text. On the basis of integrating the modification suggestions, the suggestions are comprehensively considered in combination with business goals, user needs and context; the priority and importance of each suggestion are weighed to ensure that the final modified second prompt text can accurately and efficiently guide user queries or requests.

[0109] In some implementations, the first prompt text is specifically modified and adjusted based on the comprehensive consideration results. During the modification process, the text is kept concise, clear, and coherent to ensure that the user can easily understand and follow the prompt. After the modification and adjustment of the above steps, a second prompt text that integrates multiple modification suggestions is finally generated. The second prompt text should be able to fully and accurately reflect the user's query needs, while meeting business goals and user experience requirements.

[0110] For example, suppose there is a prompt text next to the search box of an online travel booking platform to guide users to enter the destination and date. The original first prompt text is "Please enter your destination and travel date". Modification suggestions include: Suggestion 1: Add more query options, such as "or select popular destinations, recent dates". Suggestion 2: Clarify the input format, such as "Please enter in the format of 'destination, date'". Suggestion 3: Emphasize the convenience of search, such as "Easy query, fast booking". When integrating these suggestions, it is found that suggestion 1 increases the flexibility of query, suggestion 2 clarifies the input format, and suggestion 3 emphasizes the convenience of search. After comprehensive consideration, it is decided to integrate these suggestions into a concise and clear second prompt text: "Please enter your destination and travel date, or select popular destinations, recent dates (enter in the format of 'destination, date'), easy query, fast booking." Through such a comprehensive modification strategy, a comprehensive and refined second prompt text can be obtained, which can better guide users to perform query and booking operations.

[0111] In this way, by comprehensively considering multiple modification suggestions, the tedious process of trying and modifying one by one is avoided, and the modification efficiency is improved. The comprehensive modification strategy can ensure that the second prompt text is smooth and reasonable in content, logic and context, and improve the text quality. The second prompt text after comprehensive modification can more accurately guide user queries or requests, thereby improving the user experience.

[0112] In some embodiments, the first prompt text is modified based on the modification suggestion to obtain the second prompt text, including: when the first prompt text includes multiple conditions and corresponding execution content, the first prompt text is modified according to a preset modification principle, and the modification principle includes at least one of the following: adding a logical branch; merging existing statements; wherein the selection of the modification principle is determined according to the specific modification suggestion content and business needs.

[0113] In the disclosed embodiment, the logic branch refers to a conditional judgment statement or a separate guidance instruction added to the first prompt text, which is used to handle different query or request situations.

[0114] In the embodiment of the present disclosure, merging existing sentences refers to the process of integrating and simplifying repeated, redundant or similar conditions and execution contents in the first prompt text.

[0115] When processing the first prompt text containing multiple conditions and corresponding execution contents, in order to ensure that the modified second prompt text can more accurately reflect business needs and user intentions, you can follow the preset modification principles and determine which principle to select based on the specific modification suggestions and business needs.

[0116] In some embodiments, when the modification suggestion indicates that some conditions or execution contents are omitted in the first prompt text, or different guidance is required for different situations, a logic branch can be added. The logic branch can be added by adding additional conditional judgment statements, or providing separate guidance instructions for specific conditions or execution contents.

[0117] In some embodiments, when the modification suggestion indicates that there are repeated, redundant or similar conditions and execution contents in the first prompt text, the existing statements can be merged. The way to merge the statements can be to simplify the language, eliminate repeated information, or integrate multiple related conditions and execution contents into a more compact statement.

[0118] In some implementations, the intention and specific changes behind each modification suggestion are understood; the consistency of the modification suggestion with business requirements and user intentions is evaluated. According to the content of the modification suggestion and business requirements, appropriate modification principles are selected. If the modification suggestion involves the supplementation or improvement of multiple conditions and execution contents, add a logical branch. If the modification suggestion points out the existence of repeated or redundant information, merge the existing statements.

[0119] In some implementations, the first prompt text is specifically modified according to the selected modification principle. During the modification process, the clarity, coherence and logic of the text are maintained. The modified second prompt text is verified and tested to ensure that it can accurately reflect business needs and user intentions. User feedback is collected to further optimize and adjust the second prompt text.

[0120] For example, suppose there is a prompt text next to the search box of an online learning platform to guide users to choose learning courses and levels. The original first prompt text is: "Please select the course you want to study (such as mathematics, English) and level (such as elementary, intermediate)". Modification suggestions include: Suggestion 1: Add guidance for advanced courses, such as "Also includes advanced courses for you to choose." Suggestion 2: Simplify the text and merge the selection instructions for courses and levels, such as "Please select the course and level (such as elementary mathematics, advanced English)." Determine the modification principle: According to Suggestion 1, choose to add logical branches and provide separate guidance instructions for advanced courses. According to Suggestion 2, choose to merge existing sentences, simplify the text and reduce repeated information. Based on the above modification principles, the modified second prompt text is obtained: "Please select the course and level you want to study (such as mathematics elementary, English intermediate, English advanced), and other advanced courses are also available for you to choose." Or, in order to be more concise and clear, it can be further adjusted to: "Please select the course and level (mathematics elementary / English intermediate / English advanced, etc.), advanced courses are also available." Through such a modification process, a second prompt text that is both accurate and concise is obtained, which can better guide users to make choices and learn.

[0121] In this way, by adding logical branches or merging existing statements, the second prompt text can be ensured to more accurately reflect business needs and user intentions. The modified second prompt text is more concise and clear, reducing user confusion and misunderstanding, and improving user experience. By merging redundant information or simplifying language, the system can reduce the time and resource consumption of text processing and improve system efficiency.

[0122] In some embodiments, the prompt text generation method based on the large language model also includes: obtaining new output content generated by the first large language model for the second Query based on the second prompt text; obtaining a scoring score for the new output content output by the second large language model; when the scoring score reaches a second threshold, determining the second prompt text as the final prompt text for multiple first queries.

[0123] In the disclosed embodiment, the second threshold is a set minimum requirement for output content quality, and is used to determine whether the second prompt text is valid.

[0124] In some embodiments, the second prompt text is input into the first large language model, and the model is allowed to generate new output content for the second Query (which may be obtained by some transformation or refinement of the first Query) based on the second prompt text. This new output content should be closely related to the second prompt text and be able to reflect the intention of the second Query. The new output content is input into the second large language model (this model can be a model specifically used to evaluate the quality of the output content, such as a text quality assessment model based on deep learning). The second large language model will evaluate the new output content and output a rating score, which represents the quality of the new output content. If the rating score of the new output content reaches or exceeds the second threshold, then it can be considered that the second prompt text is effective and can guide the language model to generate high-quality output content.

[0125] Suppose there is an intelligent assistant on an online shopping platform that uses a language model to answer user queries. Users may enter multiple queries related to products (first query), such as "I want to buy a mobile phone with high cost performance", "Recommend a mobile phone with good camera", etc. First, try to use a prompt text (such as "Please tell me the type and characteristics of the product you want to buy") to guide the first language model to generate output content. However, this prompt text may not be specific enough, resulting in the generated output content not being accurate or relevant. Therefore, based on the user's query and feedback, the prompt text is modified and optimized to obtain the second prompt text (such as "Please tell me that the product you want to buy is a mobile phone, and indicate your requirements for cost performance or camera function"). The second prompt text is input into the first language model to generate new output content (such as "Based on your needs, I recommend you to buy a mobile phone of brand XX, which is cost-effective and has excellent camera function"). The new output content is input into the second language model for evaluation. The second language model outputs a score (such as 85 points), indicating that the new output content is of high quality. Because the score of the new output content reaches the second threshold (80 points), the second prompt text is determined as the final prompt text for multiple first queries. Through this process, it can be ensured that the answers of the intelligent assistant are more accurate and useful, and the user's shopping experience is improved.

[0126] In this way, by introducing the second language model to evaluate the output content, it can be ensured that the final selected prompt text can guide the language model to generate high-quality output content. By setting the threshold and comparing the scoring scores, the final prompt text can be selected more objectively, avoiding the interference of subjective judgment.

[0127] The schematic diagram of the architecture of the prompt text generation system based on the large language model is as follows: Figure 2 As shown, the prompt text generation system based on the large language model includes three modules: a generalized scoring agent 201, a suggestion generator 202, and an optimizer 203. These three modules work together to form a closed-loop process and automatically iterate and optimize the target prompt.

[0128] In some implementations, the universal scorer 201 is used to introduce the expected response behavior as an evaluation criterion, compare the output content of the first language model with the expected response behavior, and output an evaluation score and a detailed explanation, thereby reducing the annotation cost and improving the evaluation efficiency and flexibility.

[0129] In some embodiments, the suggestion generator 202 is used to generate general and widely applicable modification suggestions based on the low-scoring data in the output content of the first language model. Specifically, the output content of the second Query and its corresponding first language model, the expected reply behavior, the explanation output by the universal scorer 201 and the current prompt are integrated to generate general and widely applicable modification suggestions. Furthermore, multiple similar error cases are considered at the same time to generate more general modification suggestions.

[0130] In some implementations, the optimizer 203 is used to modify the current Prompt iteratively according to the modification suggestions output by the suggestion generator 202. Specifically, the combined impact of multiple modification suggestions is considered to ensure that the Prompt is optimized in the right direction. Furthermore, specific principles are followed, such as giving priority to adding logical branches or merging existing statements, and maintaining the overall structure of the Prompt.

[0131] It should be noted that the above-mentioned universal scorer 201, suggestion generator 202 and optimizer 203 can all be implemented using models. For example, the universal scorer 201 is implemented using the second largest language model, the suggestion generator 202 is implemented using the third largest language model, and the optimizer 203 is implemented using the fourth largest language model. Alternatively, the universal scorer 201, the suggestion generator 202 and the optimizer 203 are all implemented using the second largest language model. Here, the second largest language model, the third largest language model and the fourth largest language model are all pre-trained large language models.

[0132] In this way, by integrating scoring, suggestion generation and optimization modules, the manpower costs of R&D personnel and business personnel in the prompt optimization link can be significantly reduced, an efficient and low-cost prompt optimization process can be achieved, and it can adapt to the needs of different business scenarios.

[0133] The Prompt automatic optimization solution disclosed in the present invention can be widely applied to various application scenarios based on large language models. The following are several specific application examples:

[0134] Intelligent customer service system dialogue style tuning:

[0135] Application process: Collect user query samples->Define Expectation->Use a general scorer to evaluate existing responses->The suggestion generator proposes improvement suggestions->The optimizer adjusts the prompt->Iterate optimization.

[0136] Effect: Improve response accuracy and user satisfaction, and reduce manual customer service costs.

[0137] Content Generation Tools:

[0138] Application process: Collect content demand samples->Define content quality standards->Evaluate generated content->Propose improvement suggestions->Optimize prompts->Iterate to improve content quality.

[0139] Results: Improved relevance, creativity, and consistency of content generation.

[0140] Coding assistance tools:

[0141] Application process: Collect programming task samples -> define code quality standards -> evaluate generated code -> make improvement suggestions -> optimize prompts -> iteratively improve code generation quality.

[0142] Effect: Improve the correctness, efficiency and readability of generated code.

[0143] Educational guidance system:

[0144] Application process: Collect student question samples->Define teaching standards->Evaluate system answers->Propose improvement suggestions->Optimize prompts->Iterate to improve teaching quality.

[0145] Effect: Improve the pertinence of answers and teaching effectiveness to meet the needs of different students.

[0146] These application examples demonstrate the applicability and potential value of the present disclosure in different fields. Through continuous automatic optimization, the performance of large language models in various scenarios can be significantly improved. It is understood that the above application examples are merely illustrative and not restrictive, and the present disclosure does not list them exhaustively.

[0147] Figure 3 is a flowchart of optimizing prompt text based on a large language model according to an embodiment of the present disclosure, such as Figure 3As shown, the process includes: the first language model receives a series of queries, which may come from the same user or from different users; the first language model generates output content for each query based on the first prompt text; the second language model obtains the output content generated by the first language model for each query, and uses a universal scorer to generate an evaluation score and explanation for the output content of the first language model based on the expected response behavior; uses a suggestion generator to determine a second query below the first threshold from multiple first queries, and generates a modification suggestion for the first prompt text based on the expected response behavior, the second query and its corresponding output content and explanation; uses an optimizer to modify the first prompt text based on the modification suggestion to obtain an optimized prompt text (i.e., the second prompt text). The first language model generates new output content for each query based on the second prompt text, and the second language model gives a score and explanation for the new output content; when the score reaches the second threshold, the current second prompt text is determined as the final prompt text for the multiple first queries.

[0148] The Prompt automatic optimization solution disclosed in this disclosure brings the following significant benefits to products based on large language models:

[0149] 1. Reduce labor costs: Through automated processes, the manpower input of developers in the prompt tuning process can be significantly reduced.

[0150] 2. Improve optimization efficiency: The systematic optimization process can complete the tuning of a large number of prompts in a short time and accelerate product iteration.

[0151] 3. Improve model performance: Through continuous optimization, significantly improve the quality, relevance and consistency of model output.

[0152] 4. Enhanced adaptability: can quickly adapt to different business scenarios and improve the application flexibility of products in various fields.

[0153] 5. Reduce evaluation costs: Use expected response behaviors instead of standard answers to significantly reduce data annotation costs and evaluation costs.

[0154] 6. Achieve continuous optimization: Through the self-circulation mechanism, the model performance can be continuously improved.

[0155] 7. Improved explainability: The optimization process is transparent and traceable, which helps to understand and further improve the model behavior.

[0156] The use of the Prompt automatic optimization solution disclosed in the present invention will significantly improve the quality and efficiency of products based on large language models.

[0157] The present disclosure provides a device for generating prompt text based on a large language model. Figure 4 As shown, the prompt text generation device based on the large language model may include: a first acquisition unit 401, used to obtain the output content generated by the first large language model for multiple first queries based on the first prompt text; a second acquisition unit 402, used to obtain the evaluation score and explanation of the output content generated by the second large language model based on the expected response behavior; a first determination unit 403, used to determine the second Query from the multiple first Queries, the second Query is the first Query with an evaluation score lower than the first threshold; a first generation unit 404, used to generate a modification suggestion for the first prompt text based on the expected response behavior, the second Query and its corresponding output content and explanation; a second generation unit 405, used to modify the first prompt text based on the modification suggestion to obtain the second prompt text.

[0158] In some embodiments, the second acquisition unit 402 includes: a first determination subunit, used to determine the business scenario to which multiple first queries belong; a second determination subunit, used to determine the scoring criteria that match the business scenario; a control subunit, used to control the second largest language model to generate a scoring score for the output content for each first Query according to the scoring criteria; and generating an explanation of the output content for each first Query based on the expected response behavior.

[0159] In some embodiments, multiple first queries correspond to the same expected response behavior, and the expected response behavior is a response standard that is suitable for the business scenario.

[0160] In some embodiments, the first generating unit 404 is used to: when there are multiple second queries, generate modification suggestions for the first prompt text for each second query based on the output content and explanation corresponding to each second query and the expected response behavior.

[0161] In some embodiments, the first generation unit 404 is used to: when there are multiple second queries, classify the multiple second queries according to business characteristics to form multiple batches; wherein the number of second queries contained in each batch is less than the total number of second queries; for each batch, summarize the output content and explanations corresponding to the second queries in the same batch, and combine with the expected response behavior to jointly generate modification suggestions for the first prompt text corresponding to the batch.

[0162] In some embodiments, the second generation unit 405 is used to: when there are multiple modification suggestions, modify the first prompt text separately for each modification suggestion to obtain multiple candidate second prompt texts; and determine the second prompt text from the multiple candidate second prompt texts.

[0163] In some embodiments, the second generating unit 405 is used to: when there are multiple modification suggestions, modify the first prompt text by integrating the specific contents of the multiple modification suggestions to obtain the second prompt text.

[0164] In some embodiments, the second generation unit 405 is used to: when the first prompt text includes multiple conditions and corresponding execution content, modify the first prompt text according to a preset modification principle, and the modification principle includes at least one of the following: adding logical branches; merging existing statements; wherein the selection of the modification principle is determined according to the specific modification suggestion content and business needs.

[0165] In some embodiments, the first acquisition unit 401 is further used to acquire new output content generated by the first large language model for the second query based on the second prompt text; the second acquisition unit 402 is further used to acquire the score for the new output content output by the second large language model. The prompt text generation device based on the large language model may include: a second determination unit 406 ( Figure 4 (not shown) is used to determine the second prompt text as the final prompt text for multiple first queries when the scoring score reaches a second threshold.

[0166] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0167] The prompt text generation device based on the large language model of the embodiment of the present disclosure can realize an efficient and low-cost Prompt optimization process and can adapt to the needs of different business scenarios.

[0168] The present disclosure provides a scenario diagram of a method for generating prompt text based on a large language model. Figure 5 shown.

[0169] As mentioned above, the method for generating prompt text based on a large language model provided by the embodiment of the present disclosure is applied to electronic devices. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices.

[0170] Specifically, the electronic device can perform the following operations:

[0171] Obtain output content generated by the first language model for the plurality of first queries based on the first prompt text;

[0172] Obtain the evaluation score and explanation of the output content generated by the second largest language model based on the expected response behavior;

[0173] Determine a second query from the plurality of first queries, the second query being a first query whose evaluation score is lower than a first threshold;

[0174] Generate modification suggestions for the first prompt text based on the expected response behavior, the second Query and its corresponding output content and explanation;

[0175] The first prompt text is modified based on the modification suggestion to obtain a second prompt text.

[0176] It should be understood that Figure 5 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Figure 5 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0177] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0178] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0179] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0180] like Figure 6As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0181] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0182] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a prompt text generation method based on a large language model. For example, in some embodiments, the prompt text generation method based on a large language model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the prompt text generation method based on a large language model described above may be executed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the prompt text generation method based on a large language model in any other appropriate manner (for example, by means of firmware).

[0183] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0184] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0185] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0186] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0187] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0188] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0189] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0190] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating prompt text based on a large language model, comprising: Obtain output content generated by the first language model for multiple first queries based on the first prompt text; Obtaining an evaluation score and an explanation of the output content generated by a second language model based on the expected response behavior; Determine a second query from the plurality of first queries, the second query being a first query having an evaluation score lower than a first threshold; Based on the expected response behavior, the second Query and its corresponding output content and explanation, generating a modification suggestion for the first prompt text; The first prompt text is modified based on the modification suggestion to obtain a second prompt text.

2. The method according to claim 1, wherein: The obtaining of an evaluation score and an explanation of the output content generated by the second largest language model based on the expected response behavior comprises: Determine the business scenarios to which multiple first queries belong; Determine the scoring criteria that match the business scenario; Control the second largest language model to generate a scoring score for the output content for each first Query according to the scoring standard; and generate an explanation of the output content for each first Query based on the expected response behavior.

3. The method according to claim 2, wherein: The multiple first queries correspond to the same expected response behavior, and the expected response behavior is a response standard that is suitable for the business scenario.

4. The method according to claim 1, wherein: The generating a modification suggestion for the first prompt text based on the expected response behavior, the second Query and the corresponding output content and explanation thereof includes: In the case where there are multiple second queries, a modification suggestion for the first prompt text is generated for each second query based on the output content and explanation corresponding to each second query and the expected response behavior.

5. The method according to claim 1, wherein: The generating a modification suggestion for the first prompt text based on the expected response behavior, the second Query and the corresponding output content and explanation thereof includes: In the case where there are multiple second queries, the multiple second queries are classified according to business characteristics to form multiple batches; wherein the number of second queries included in each batch is less than the total number of second queries; For each batch, the output content and explanation corresponding to the second query in the same batch are summarized, and combined with the expected response behavior, a modification suggestion for the first prompt text corresponding to the batch is jointly generated.

6. The method according to claim 1, wherein: The step of modifying the first prompt text based on the modification suggestion to obtain a second prompt text includes: When there are multiple modification suggestions, modify the first prompt text separately for each modification suggestion to obtain multiple candidate second prompt texts; The second prompt text is determined from the multiple candidate second prompt texts.

7. The method according to claim 1, wherein: The step of modifying the first prompt text based on the modification suggestion to obtain a second prompt text includes: When there are multiple modification suggestions, the first prompt text is modified based on the specific contents of the multiple modification suggestions to obtain the second prompt text.

8. The method according to claim 1, wherein: The step of modifying the first prompt text based on the modification suggestion to obtain a second prompt text includes: In the case where the first prompt text includes multiple conditions and corresponding execution contents, the first prompt text is modified according to preset modification principles, and the modification principles include at least one of the following: adding logical branches; merging existing statements; wherein the selection of the modification principles is determined according to the specific modification suggestion content and business requirements.

9. The method according to claim 1, further comprising: Obtain new output content generated by the first language model for the second Query based on the second prompt text; Obtaining a scoring score for the new output content output by the second largest language model; When the scoring score reaches a second threshold, the second prompt text is determined as the final prompt text for the multiple first queries.

10. A prompt text generation device based on a large language model, comprising: A first acquisition unit, configured to acquire output content generated by a first language model for a plurality of first queries based on a first prompt text; A second acquisition unit is used to acquire an evaluation score and an explanation of the output content generated by a second language model based on the expected response behavior; A first determining unit, configured to determine a second query from a plurality of first queries, wherein the second query is a first query having an evaluation score lower than a first threshold; A first generating unit, configured to generate a modification suggestion for the first prompt text based on the expected response behavior, the second Query and its corresponding output content and explanation; The second generating unit is used to modify the first prompt text based on the modification suggestion to obtain a second prompt text.

11. The device according to claim 10, wherein: The second acquisition unit includes: A first determination subunit, used to determine the business scenarios to which the multiple first queries belong; A second determination subunit, used to determine a scoring standard matching the business scenario; A control subunit is used to control the second largest language model to generate a scoring score for the output content for each first Query according to the scoring standard; and to generate an explanation of the output content for each first Query based on the expected response behavior.

12. The device according to claim 11, wherein The multiple first queries correspond to the same expected response behavior, and the expected response behavior is a response standard that is suitable for the business scenario.

13. The device according to claim 10, wherein: The first generation unit is configured to: In the case where there are multiple second queries, a modification suggestion for the first prompt text is generated for each second query based on the output content and explanation corresponding to each second query and the expected response behavior.

14. The device according to claim 10, wherein: The first generation unit is configured to: In the case where there are multiple second queries, the multiple second queries are classified according to business characteristics to form multiple batches; wherein the number of second queries included in each batch is less than the total number of second queries; For each batch, the output content and explanation corresponding to the second query in the same batch are summarized, and combined with the expected response behavior, a modification suggestion for the first prompt text corresponding to the batch is jointly generated.

15. The device according to claim 10, wherein: The second generating unit is used for: When there are multiple modification suggestions, modify the first prompt text separately for each modification suggestion to obtain multiple candidate second prompt texts; The second prompt text is determined from the multiple candidate second prompt texts.

16. The device according to claim 10, wherein: The second generating unit is used for: When there are multiple modification suggestions, the first prompt text is modified based on the specific contents of the multiple modification suggestions to obtain the second prompt text.

17. The device according to claim 10, wherein: The second generating unit is used for: In the case where the first prompt text includes multiple conditions and corresponding execution contents, the first prompt text is modified according to a preset modification principle, and the modification principle includes at least one of the following: adding a logic branch; Merge existing statements; wherein the selection of the modification principle is determined according to the specific modification suggestion content and business requirements.

18. The device according to claim 10, wherein: The first acquisition unit is further used to acquire new output content generated by the first language model for the second Query based on the second prompt text; the second acquisition unit is further used to acquire a score for the new output content output by the second language model; The device also includes: The second determining unit is configured to determine the second prompt text as the final prompt text for the multiple first queries when the scoring score reaches a second threshold.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are for causing a computer to execute a method according to any one of claims 1-9.

21. A computer program product comprising a computer program stored on a storage medium, the computer program implementing the method according to any one of claims 1 to 9 when executed by a processor.

Images