Prompt text generation method and device based on large language model, and storage medium

Abstract

The present disclosure provides a large language model-based prompt text generation method and device and storage medium. The present disclosure relates to the field of artificial intelligence technology, in particular to the field of intelligent search, large model and prompt text optimization, and can be applied to application scenarios such as intelligent customer service systems, automatic question and answer systems, content generation tools, code auxiliary tools, data analysis assistants, education guidance systems, medical diagnosis assistance systems and the like. The specific implementation scheme is as follows: obtaining output content generated by a first large language model based on a first prompt text for a plurality of first queries; obtaining an evaluation score and an explanation of the output content generated by a second large language model based on an expected response behavior; determining a second query from the plurality of first queries; generating a modification suggestion for the first prompt text based on the expected response behavior, the second query, the corresponding output content and the explanation; and modifying the first prompt text based on the modification suggestion to obtain a second prompt text.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the fields of intelligent search, large models, and prompt text optimization. The present disclosure can be used in application scenarios such as intelligent customer service systems, automatic question-and-answer systems, content generation tools, code assistance tools, data analysis assistants, educational tutoring systems, and medical diagnosis assistance systems. The present disclosure specifically relates to a prompt text generation method, device, and storage medium based on a large language model. Background Art

[0002] With advances in machine learning, researchers have attempted to automate the optimization of Prompt. For example, some have proposed gradient-based search methods to optimize Prompt. However, these methods still require significant computing resources and data, making them difficult to adapt quickly and efficiently in complex and diverse scenarios. This is particularly true in scenarios that require processing large amounts of unstructured input, such as intelligent customer service and automated question-and-answering. Summary of the Invention

[0003] The present disclosure provides a method, device, and storage medium for generating prompt text based on a large language model.

[0004] According to one aspect of the present disclosure, a method for generating prompt text based on a large language model is provided, the method comprising:

[0005] Obtain output content generated by the first language model for multiple first queries based on the first prompt text;

[0006] Obtain the evaluation score and explanation of the output content generated by the second largest language model based on the expected response behavior;

[0007] Determine a second query from the plurality of first queries, the second query being a first query having an evaluation score lower than a first threshold;

[0008] Generate modification suggestions for the first prompt text based on the expected response behavior, the second query and its corresponding output content and explanation;

[0009] The first prompt text is modified based on the modification suggestion to obtain a second prompt text.

[0010] According to another aspect of the present disclosure, a device for generating prompt text based on a large language model is provided, the device comprising:

[0011] A first acquisition unit is configured to acquire output content generated by a first language model for a plurality of first queries based on the first prompt text;

[0012] a second acquisition unit, configured to acquire an evaluation score and an explanation of the output content generated by the second language model based on the expected response behavior;

[0013] A first determining unit is configured to determine a second query from a plurality of first queries, where the second query is a first query having an evaluation score lower than a first threshold;

[0014] A first generating unit, configured to generate a modification suggestion for the first prompt text based on the expected response behavior, the second query and its corresponding output content and explanation;

[0015] The second generating unit is configured to modify the first prompt text based on the modification suggestion to obtain a second prompt text.

[0016] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any method in the embodiments of the present disclosure.

[0020] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0021] According to another aspect of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the computer program implements any one of the methods according to the embodiments of the present disclosure.

[0022] By adopting the solution disclosed in the present invention, Prompt can be automatically optimized in a systematic and low-cost manner, and can adapt to the needs of different business scenarios.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0025] Figure 1 is a flowchart of a method for generating prompt text based on a large language model according to an embodiment of the present disclosure;

[0026] Figure 21 is a schematic diagram of the architecture of a prompt text generation system based on a large language model according to an embodiment of the present disclosure;

[0027] Figure 3 is a flow chart of optimizing prompt text based on a large language model according to an embodiment of the present disclosure;

[0028] Figure 4 is a structural diagram of a prompt text generation device based on a large language model according to an embodiment of the present disclosure;

[0029] Figure 5 1 is a schematic diagram of a scenario of a method for generating prompt text based on a large language model according to an embodiment of the present disclosure;

[0030] Figure 6 3 is a schematic diagram of the structure of an electronic device used to implement the method for generating prompt text based on a large language model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them, and do not mean to limit the order or to limit to only two. For example, the first feature and the second feature refer to two categories / two features. The first feature can be one or more, and the second feature can also be one or more.

[0033] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0034] Before introducing the technical solutions of the embodiments of the present disclosure, the following technical terms that may be used in the present disclosure are further explained:

[0035] Large language model: refers to a model with large-scale parameters and training data that can generate coherent and natural language.

[0036] Query: Typically, it is text or instructions input to a large language model, guiding the model to generate the appropriate output. Users submit queries to request the model to generate tasks such as answers, text generation, translation, and summarization. Queries can range from simple sentences and paragraphs to more complex instructions, depending on the task the user wants the model to complete.

[0037] Expected response behavior (Expect): refers to the standard or pattern that the output content generated by the large language model should meet.

[0038] Suggestion: A specific suggestion for improving the prompt text.

[0039] In related technologies, the prompt optimization technology has the following main disadvantages:

[0040] 1. High labor costs: Traditional manual trial and error methods require a lot of manpower input. Especially in complex scenarios, the optimization process may take days or even weeks, greatly increasing project costs.

[0041] 2. Lack of systematicity: Existing methods often lack a systematic optimization process, resulting in unstable optimization results and difficulty in replicating and extending them to other scenarios.

[0042] 3. Poor adaptability: Template-based and rule-based methods are difficult to adapt to novel or complex business scenarios, limiting the application scope of large language models.

[0043] 4. High evaluation cost: The evaluation method based on the Golden Answer requires labeling the standard answer for each query, which is unrealistic in large-scale applications and cannot meet the needs of rapid iteration.

[0044] 5. Unstable optimization results: Although machine learning optimization methods have certain adaptability, they often require a large amount of training data and tuning, resulting in unstable optimization results and difficulty in ensuring consistency.

[0045] 6. Difficulty handling complex tasks: Existing methods perform poorly when handling tasks that require multi-step reasoning or complex logic, making it difficult to effectively optimize Prompt to improve model performance.

[0046] 7. Lack of feedback loop: Most existing methods lack an effective feedback mechanism, making it difficult to dynamically adjust optimization strategies based on model output and actual results.

[0047] The above shortcomings severely limit the effectiveness and efficiency of large language models in practical applications. There is an urgent need for an automated, systematic, low-cost, and highly adaptable (universal) Prompt optimization solution to solve these problems.

[0048] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, the present disclosure proposes a prompt text generation method based on a large language model. The method can realize automatic optimization of prompts through a universal scorer, suggestion generator and optimizer, significantly reduce the labor cost in the prompt optimization link, realize an efficient and low-cost prompt optimization process, and can adapt to the needs of different business scenarios.

[0049] The present disclosure provides a method for generating prompt text based on a large language model. Figure 1 This is a flow chart of a prompt text generation method based on a large language model according to an embodiment of the present disclosure. The prompt text generation method based on a large language model can be applied to a prompt text generation device based on a large language model. The prompt text generation device based on a large language model is located in an electronic device. The electronic device includes but is not limited to fixed devices and / or mobile devices. For example, fixed devices include but are not limited to servers, and servers can be cloud servers or ordinary servers. For example, mobile devices include but are not limited to: mobile phones, tablet computers. In some possible implementations, the prompt text generation method based on a large language model can also be implemented by a processor calling computer-readable instructions stored in a memory. For example Figure 1 As shown, the prompt text generation method based on the large language model includes:

[0050] S101, obtaining output content generated by a first language model for multiple first queries based on a first prompt text;

[0051] S102. Obtain an evaluation score and explanation of the output content generated by the second language model based on the expected response behavior;

[0052] S103, determining a second query from the plurality of first queries, where the second query is a first query whose evaluation score is lower than the first threshold;

[0053] S104: Generate modification suggestions for the first prompt text based on the expected response behavior, the second query and its corresponding output content and explanation;

[0054] S105: Modify the first prompt text based on the modification suggestion to obtain a second prompt text.

[0055] In the embodiment of the present disclosure, the first prompt text is a prompt text for multiple first queries at a first moment, and the second prompt text is a prompt text for multiple first queries at a second moment, where the second moment is later than the first moment.

[0056] In the embodiment of the present disclosure, the second prompt text is a revised version based on the modification suggestions for the first prompt text. Compared with the first prompt text, the second prompt text shows better performance in terms of applicability.

[0057] In the disclosed embodiments, the first large language model and the second large language model can be different large language models or the same large language model, i.e., a large language model is used to both generate results based on the query output and output scores and explanations for the generated results based on expected response behavior. The first and second language models include, but are not limited to, pre-trained language generation models, language models based on the Transformer architecture, or language generation models optimized for specific domains.

[0058] In the embodiment of the present disclosure, the expected response behavior is to evaluate whether the output content conforms to a certain specific, expected response mode or standard.

[0059] In the disclosed embodiment, the evaluation score is determined by the second largest language model in combination with the expected reply behavior and according to a preset scoring standard, and the preset scoring standard is adapted to the business scenario to which the Query belongs.

[0060] In some embodiments, obtaining output content generated by the first large language model for multiple first queries based on the first prompt text includes: determining an initial prompt text, which is the starting point for the large language model to generate corresponding content based on it; using the first large language model to generate output content for multiple different first queries (these queries can be questions, instructions or topics related to the first prompt text) according to the first prompt text, and these output contents will serve as the basis for subsequent evaluation.

[0061] In some embodiments, obtaining an evaluation score and explanation of the output content generated by the second largest language model based on the expected response behavior includes: using the second largest language model (which can be another trained large language model or a language model specifically used for evaluation) to evaluate the output content generated by the first largest language model, and the second largest language model generates an evaluation score for each output content, as well as an explanation of why this score is given.

[0062] In some embodiments, determining a second query from multiple first queries includes: screening out first queries whose evaluation scores are lower than a first threshold based on the evaluation scores given by the second largest language model, and the output content corresponding to these queries may not meet the expected response behavior or have low quality, and marking these screened queries as second queries as a reference for subsequent modification of the prompt text.

[0063] In some embodiments, generating modification suggestions for the first prompt text based on the expected response behavior, the second query, and its corresponding output content and explanation includes analyzing the explanation provided by the second language model to understand why the output content corresponding to the second query does not conform to the expected response behavior. Based on this analysis, generating modification suggestions for the first prompt text. These suggestions are intended to improve the prompt text so that it can guide the first language model to generate output content that is more consistent with the expected response behavior.

[0064] In some embodiments, the first prompt text is modified based on the modification suggestion to obtain the second prompt text, including: modifying the first prompt text according to the generated modification suggestion; the modified text is the second prompt text, which should be able to guide the first language model to generate output content with higher quality and more consistent with the expected response behavior.

[0065] For example, suppose the first prompt is "Please write an article about the development of artificial intelligence." Based on this prompt, the first language model generates output content for multiple queries (such as "the history of artificial intelligence," "the current state of artificial intelligence," and "the future of artificial intelligence"). The second language model then evaluates these outputs based on expected response behavior (such as clear structure, accurate content, and fluent language), providing scores and explanations. Suppose the output content for the query "the future of artificial intelligence" receives a low evaluation score because it is too abstract and lacks specific examples. Based on this evaluation result, a modification suggestion for the first prompt can be generated, such as "Add requirements for specific content and examples to the prompt." Finally, the first prompt is modified based on this modification suggestion, resulting in a second prompt text, such as "Please write an article about the development of artificial intelligence, including specific historical events, current application examples, and a vision for the future." This approach allows for continuous optimization of the prompt text, improving the quality and accuracy of the text generated by the large language model.

[0066] According to the technical solution of the embodiment of the present disclosure, the second largest language model evaluates the output content of the first largest language model based on the expected response behavior to obtain an evaluation score and explanation; based on the expected response behavior, the second query with an evaluation score lower than the first threshold and its corresponding output content and explanation, a modification suggestion for the first prompt text is generated; based on the modification suggestion, the first prompt text is modified to obtain the second prompt text; in this way, the prompt can be optimized in an automated, systematic, low-cost and highly adaptable manner, thereby improving the quality and efficiency of products based on the large language model. Since there is no need to generate a standard answer for each query, the expected response behavior is used instead of the standard answer, which not only greatly reduces the data annotation cost, but also significantly reduces the manpower investment in the prompt tuning link and reduces the evaluation cost. The systematic optimization process can complete the tuning of a large number of prompts in a short period of time, accelerating product iteration; through continuous optimization, the model performance is continuously improved, the quality, relevance and consistency of the model output are significantly improved, and it can quickly adapt to different business scenarios and improve the application flexibility of the product in various fields. Since the optimization process is transparent and traceable, it helps to understand and further improve the model behavior and provide optimization explainability.

[0067] In some embodiments, obtaining an evaluation score and explanation of the output content generated by the second largest language model based on the expected response behavior includes: determining the business scenario to which the multiple first queries belong; determining a scoring criterion that matches the business scenario; generating a scoring score for the output content for each first query according to the scoring criterion; and generating an explanation of the output content for each first query based on the expected response behavior.

[0068] In the disclosed embodiment, the second largest language model is a model used to evaluate the output content. It may not be the most advanced or complex model, but performs well in specific tasks or scenarios.

[0069] In the embodiments of the present disclosure, the expected response behavior is the response standard that the user expects to obtain from the system or model in a given business scenario.

[0070] In some implementations, the business scenario to which each first query (ie, user input or request) belongs is clearly defined. Here, business scenarios include, but are not limited to, customer service consultation, product recommendations, technical support, and problem solving.

[0071] In some implementations, a matching scoring criterion is selected based on the identified business scenario. The scoring criterion may include multiple dimensions such as accuracy, relevance, timeliness, and friendliness. Different business scenarios may require different scoring criterion combinations or weightings.

[0072] In some implementations, for each first query, the second language model scores its output content based on selected scoring criteria. This score reflects the performance of the output content across multiple dimensions, such as whether it accurately answers the user's question, is highly relevant to the user's needs, and is timely.

[0073] In some embodiments, in addition to the scoring, the second language model also generates explanations for each query based on expected response behavior. These explanations provide the rationale behind the score, helping users or system developers understand why the output received a certain score. For example, if the output received a low score for accuracy, the explanation might point out errors or omissions.

[0074] For example, imagine a user enters a query regarding product returns and exchanges into the customer service system of an e-commerce platform. The second language model first determines that this query falls into the "customer service consultation" business scenario and then matches it with relevant scoring criteria, such as "accuracy," "friendliness," and "timeliness." The model then scores the system's response based on these criteria and generates an explanation explaining why the response scored highly for "accuracy" but needs improvement in "friendliness." This allows system developers to optimize responses based on this feedback.

[0075] In this way, by matching business scenarios and scoring criteria, the second largest language model can more accurately evaluate the quality of the output content and improve the accuracy of the evaluation; providing explanations not only increases the transparency of the evaluation process, but also helps system developers understand how to improve the output content. This method can be integrated into automated systems to achieve real-time evaluation and feedback on the output content.

[0076] In some embodiments, multiple first queries correspond to the same expected response behavior, and the expected response behavior is a response standard that is adapted to the business scenario.

[0077] In the embodiment of the present disclosure, the first Query is a request or question raised by a user or a system to the first language model.

[0078] In the embodiments of the present disclosure, a business scenario is the specific environment and background in which a user or system uses a model or service, which determines the specific content and requirements of the expected response behavior.

[0079] In the disclosed embodiment, although multiple first queries may involve different specific topics or details, they all correspond to the same expected response behavior. This expected response behavior is determined based on the business scenario and represents the response standard expected by the user or system in that scenario.

[0080] In some implementations, the specific type and characteristics of the business scenario are identified, such as product inquiries on e-commerce platforms, financial services inquiries at banks, or health inquiries at medical institutions. Based on the characteristics of the business scenario, the corresponding expected response behavior is defined. This behavior may include multiple aspects such as timeliness, accuracy, professionalism, and friendliness of the response.

[0081] In some implementations, multiple first queries are designed based on business scenarios and expected response behaviors. These queries cover various questions or requests that users may raise, while ensuring that they are all relevant to the expected response behaviors.

[0082] In some embodiments, a first language model is used to generate output content for each first query based on the first prompt text. A second language model is used to evaluate the generated output content based on the expected response behavior (here specifically referring to the response standard that is appropriate to the business scenario), and a score and explanation are given. Based on the evaluation results of the second language model, an analysis is performed to determine which query output content meets the expected response behavior, which does not, and the reasons for the non-compliance. Based on the evaluation results and analysis, the first prompt text is modified and optimized to improve the quality of the output content generated by the first language model so that it better meets the expected response behavior.

[0083] For example, consider the customer service system of an online education platform, where the business scenario is "course consultation." In this scenario, expected responses might include "responding to user questions promptly," "providing accurate course information," and "answering user questions in a friendly manner." Based on this business scenario and expected response, multiple first queries can be designed, such as "What are the class times for this course?", "Who teaches this course?", and "Who is this course suitable for?" The first language model is then used to generate output for each query, and the second language model is used to evaluate the output based on the expected response. If the output of a query scores low for "providing accurate course information," the first prompt text can be modified and optimized to improve the quality of the generated output.

[0084] By clarifying the expected response behavior and adapting it to the business scenario, we can ensure that the generated output content better meets user or system expectations, thereby improving response quality. Having multiple first queries corresponding to the same expected response behavior helps ensure that the output content generated across different topics or details remains consistent and coherent overall. High-quality responses and consistent output content can improve user satisfaction with the system or service, thereby optimizing the user experience.

[0085] In some embodiments, based on the expected response behavior, the second query and its corresponding output content and explanation, modification suggestions for the first prompt text are generated, including: in the case of multiple second queries, for the output content and explanation corresponding to each second query, combined with the expected response behavior, modification suggestions for the first prompt text are generated for each second query.

[0086] In the embodiment of the present disclosure, the second query is the first query whose evaluation score is lower than the first threshold, that is, the query whose generated output content does not meet the expected response behavior or is of low quality.

[0087] In the embodiment of the present disclosure, the modification suggestion is a specific suggestion for improving the first prompt text based on the output content and explanation and the expected response behavior.

[0088] In some embodiments, when there are multiple second queries, the output content and explanation of each second query are analyzed; for each second query, its corresponding output content and the explanation given by the second largest language model are carefully analyzed; the aspects in which the output content does not conform to the expected response behavior, as well as the specific problems or defects mentioned in the explanation, are understood, and modification suggestions for the first prompt text are generated.

[0089] In some embodiments, the output content and explanation of each second query are compared with the expected response behavior. It is determined which deficiencies lead to the output content not meeting expectations, such as the accuracy, completeness, relevance of the information, or clarity of expression. Based on the above analysis, modification suggestions for the first prompt text are generated for each second query. These suggestions should be aimed at improving the prompt text so that it can guide the first language model to generate output content that is more consistent with the expected response behavior.

[0090] In some embodiments, modification suggestions may include adding specific information, adjusting the language style, clarifying instruction requirements, etc. For example, if the output content has issues with accuracy, you may suggest adding more specific details or restrictions to the prompt text; if the output content lacks clarity, you may suggest adjusting the language structure or adding explanatory statements.

[0091] In some implementations, after generating modification suggestions for the first prompt text for each second query, these modification suggestions are comprehensively evaluated to ensure that the modification suggestions not only solve the current problem but also do not introduce new deficiencies or contradictions. The feasibility and implementation cost of the modification suggestions are also considered.

[0092] In some implementations, based on the results of a comprehensive evaluation of the modification suggestions, the modification suggestions are optimized and iterated to ensure that the final modification suggestions are both effective and practical.

[0093] For example, in a travel planning service, the business scenario is "making a travel plan." Expected responses might include "providing a detailed itinerary," "including necessary transportation and accommodation information," and "ensuring the feasibility and safety of the plan." Among multiple first queries, two second queries received low evaluation scores. The first second query, "How do I plan a trip from Shanghai to Beijing?", lacked the "providing a detailed itinerary" feature. The second second query, "I need a travel plan suitable for a family trip," lacked the "inclusion of necessary transportation and accommodation information" feature. For these two second queries, the following modification suggestions can be generated: For the first second query, the suggestion is to add "Please provide a detailed itinerary from the departure point to the destination, including information on the duration, transportation method, and route for each attraction." For the second second query, the suggestion is to add "Please create a travel plan for a family trip that includes necessary transportation and accommodation information, ensuring the comfort and safety of the plan." By implementing these modification suggestions, the first language model can be guided to better meet user expectations and needs when generating output.

[0094] By generating specific modification suggestions for each second query, we can more effectively improve the first prompt text, thereby enhancing the quality of the output content generated by the first language model. Each modification suggestion is generated based on the specific output content and explanation, as well as the expected response behavior, making it highly targeted and practical. Based on the generated modification suggestions, the prompt text is continuously analyzed and improved, driving the continuous optimization and progress of the first language model in generating output content.

[0095] In some embodiments, based on the expected response behavior, the second query and its corresponding output content and explanation, a modification suggestion for the first prompt text is generated, including: when there are multiple second queries, the multiple second queries are classified according to business characteristics to form multiple batches; wherein the number of second queries contained in each batch is less than the total number of second queries; for each batch, the output content and explanation corresponding to the second queries in the same batch are summarized, and combined with the expected response behavior, a modification suggestion for the first prompt text corresponding to the batch is jointly generated.

[0096] In some embodiments, first, the plurality of second Queries are classified according to business characteristics or similarity of query content. The classification can be based on the subject matter, type, priority of the query, etc. Then, the classified second Queries are divided into batches, and the number of second Queries included in each batch should be less than the total number of second Queries to ensure that the processing of each batch is more detailed and efficient.

[0097] In some embodiments, for each batch, the output content and explanations corresponding to all second Queries in the batch are collected. The output content is usually a direct answer or processing result to the query, while the explanation is a detailed description or basis for the output content. These output content and explanations are summarized to form a comprehensive understanding and evaluation of the batch of queries.

[0098] In some embodiments, based on the summarized output content and explanations, and the expected response behavior (i.e., how the system or human is expected to respond to these queries), the deficiencies of the current first prompt text in guiding queries, providing information or solving problems are analyzed. Based on these analyses, modification suggestions for the first prompt text corresponding to the batch are generated collectively. The modification suggestions may include adjusting the wording of the prompt text, adding or deleting certain information, changing the order or structure of the prompt, etc.

[0099] Suppose a query system of an e-commerce platform, users can find product information by inputting queries. The user-input queries are classified according to product categories (such as electronics, clothing, home furnishings, etc.), and then each category is processed as a batch. For queries in the electronics category, all query results and explanations about electronics are collected, such as price, function, evaluation, etc. If it is found that users often feel confused about the price range and function characteristics when searching for electronics, then guidance information about the price range and function characteristics can be added to the first prompt text to better meet the user's query needs. Through such an implementation, the performance and user experience of the query system can be continuously optimized.

[0100] In this way, by classifying and processing multiple second Queries in batches, a large number of queries can be processed in a more orderly and efficient manner, avoiding confusion and omissions in the processing process. By summarizing the output content and explanations of second Queries in the same batch and analyzing in combination with the expected response behavior, the problems in the first prompt text can be more accurately identified, and targeted modification suggestions can be generated. The optimized first prompt text can more accurately guide user queries, provide more relevant information or solutions, thereby improving user satisfaction.

[0101] In some embodiments, the first prompt text is modified based on the modification suggestion to obtain the second prompt text, including: when there are multiple modification suggestions, the first prompt text is modified separately for each modification suggestion to obtain multiple candidate second prompt texts; and the second prompt text is determined from the multiple candidate second prompt texts.

[0102] In the embodiment of the present disclosure, the candidate second prompt texts are multiple candidate texts obtained by modifying the first prompt text according to the modification suggestions.

[0103] In some embodiments, after obtaining modification suggestions for the first prompt text, in order to ensure the effectiveness and accuracy of the modifications, when there are multiple modification suggestions, each suggestion is analyzed one by one and applied to the first prompt text respectively. This means that for each modification suggestion, a modified candidate second prompt text will be obtained. When applying the modification suggestions, it is necessary to ensure that the modified text is grammatically, logically and contextually smooth and reasonable. Through the above steps, multiple candidate second prompt texts based on different modification suggestions will be obtained. These texts may differ in content, but they are all intended to optimize the first prompt text to better guide user queries or requests.

[0104] In some embodiments, after obtaining multiple candidate second prompt texts, their effectiveness needs to be further evaluated. This can be done through user testing, expert review, or data analysis. Evaluation criteria can include clarity, guidance, user satisfaction, etc. Based on the evaluation results, the best performing candidate text is selected as the final second prompt text.

[0105] For example, suppose an online shopping platform has a prompt next to the search box to guide users in entering their search query. The original first prompt was "Please enter the name of the product you want to purchase." Suggested modifications include: Suggestion 1: Add a search scope prompt, such as "Enter the product name, brand, or keyword in the search box." Suggestion 2: Emphasize search convenience, such as "Quickly search for the product you want. Type to search." Suggestion 3: Add promotional information, such as "Search for products and enjoy limited-time discounts." Candidates for the second prompt include: Candidate 1: "Enter the product name, brand, or keyword in the search box." Candidate 2: "Quickly search for the product you want. Type to search." Candidate 3: "Search for products and enjoy limited-time discounts." Testing and data analysis revealed that Candidate 1 performed best in guiding users to clearly enter their search terms, so it was selected as the final second prompt. This implementation ensures that the second prompt meets business requirements while effectively improving the user experience.

[0106] By applying each suggested change individually, multiple candidate texts can be generated, allowing the optimal change to be selected, improving the accuracy of the change. After generating multiple candidate texts, flexible selection can be made based on user feedback or business needs, making the second prompt text more adaptable to different scenarios and user needs. This optimized second prompt text can more accurately guide user queries or requests, thereby improving the user experience.

[0107] In some embodiments, the first prompt text is modified based on the modification suggestion to obtain the second prompt text, including: when there are multiple modification suggestions, the specific contents of the multiple modification suggestions are integrated to modify the first prompt text to obtain the second prompt text.

[0108] In some implementations, the intent and specific changes behind each suggested change are understood; similar, complementary, or duplicate suggestions are categorized and consolidated to avoid redundant or contradictory information in the second prompt text. Based on the integration of suggested changes, the suggestions are comprehensively considered in light of business objectives, user needs, and context; the priority and importance of each suggestion are weighed to ensure that the final revised second prompt text can accurately and efficiently guide the user's query or request.

[0109] In some implementations, specific modifications and adjustments are made to the first prompt text based on the comprehensive considerations. During this modification process, the text is maintained for simplicity, clarity, and coherence to ensure that users can easily understand and follow the prompt. After the modifications and adjustments described above, a second prompt text is ultimately generated that incorporates the multiple suggested modifications. This second prompt text should fully and accurately reflect the user's query needs while meeting business objectives and user experience requirements.

[0110] For example, consider an online travel booking platform with a prompt next to the search box, guiding users to enter their destination and dates. The original first prompt reads, "Please enter your destination and travel dates." Suggested modifications include: 1: Add more search options, such as "Or select popular destinations or recent dates." 2: Clarify the input format, such as "Please enter in the 'destination, date' format." 3: Emphasize search convenience, such as "Easy search, fast booking." When integrating these suggestions, it was found that 1 increased search flexibility, 2 clarified the input format, and 3 emphasized search convenience. After comprehensive consideration, the decision was made to combine these suggestions into a concise second prompt: "Please enter your destination and travel dates, or select popular destinations or recent dates (enter in the 'destination, date' format) for easy search and fast booking." This comprehensive modification strategy resulted in a comprehensive yet concise second prompt that better guided users through search and booking operations.

[0111] By comprehensively considering multiple suggested changes, we avoid the tedious process of trying and revising each one individually, improving modification efficiency. This comprehensive modification strategy ensures that the second prompt is coherent and reasonable in terms of content, logic, and context, improving the quality of the text. This comprehensively revised second prompt can more accurately guide user queries or requests, thereby enhancing the user experience.

[0112] In some embodiments, the first prompt text is modified based on the modification suggestion to obtain the second prompt text, including: when the first prompt text includes multiple conditions and corresponding execution content, the first prompt text is modified according to a preset modification principle, and the modification principle includes at least one of the following: adding a logical branch; merging existing statements; wherein, the selection of the modification principle is determined according to the specific modification suggestion content and business needs.

[0113] In the embodiment of the present disclosure, the logic branch refers to a conditional judgment statement or a separate guidance instruction added to the first prompt text, which is used to handle different query or request situations.

[0114] In the embodiment of the present disclosure, merging existing statements refers to the process of integrating and simplifying repeated, redundant or similar conditions and execution contents in the first prompt text.

[0115] When processing the first prompt text containing multiple conditions and corresponding execution content, in order to ensure that the modified second prompt text can more accurately reflect business needs and user intentions, you can follow the preset modification principles and determine which principle to choose based on the specific modification suggestions and business needs.

[0116] In some embodiments, when the modification suggestion indicates that certain conditions or execution contents in the first prompt text are omitted, or different guidance is required for different situations, a logical branch can be added. The logical branch can be added by adding additional conditional judgment statements or providing separate guidance instructions for specific conditions or execution contents.

[0117] In some embodiments, when the modification suggestion indicates that the first prompt text contains repeated, redundant, or similar conditions and execution contents, an option may be provided to merge the existing statements. This may be done by simplifying the language, eliminating duplicate information, or integrating multiple related conditions and execution contents into a more compact statement.

[0118] In some implementations, the intent and specific changes behind each suggested change are understood; the suggested change is evaluated for alignment with business requirements and user intent. Based on the suggested change and business requirements, appropriate modification principles are selected. If the suggested change involves multiple conditions and requires additional or improved execution content, a logical branch is added. If the suggested change indicates duplicate or redundant information, existing statements are merged.

[0119] In some implementations, specific modifications are made to the first prompt text based on the selected modification principles. During the modification process, the text's clarity, coherence, and logic are maintained. The modified second prompt text is verified and tested to ensure that it accurately reflects business needs and user intent. User feedback is collected to further optimize and adjust the second prompt text.

[0120] For example, suppose there is a prompt text next to the search box of an online learning platform to guide users to choose learning courses and levels. The original first prompt text is: "Please select the course you want to study (such as mathematics, English) and level (such as elementary, intermediate)." Modification suggestions include: Suggestion 1: Add guidance for advanced courses, such as "There are also advanced courses for you to choose from." Suggestion 2: Simplify the text and merge the selection instructions for courses and levels, such as "Please select the course and level (such as elementary mathematics, advanced English)." Determine the modification principle: According to Suggestion 1, choose to add logical branches and provide separate guidance instructions for advanced courses. According to Suggestion 2, choose to merge existing sentences, simplify the text and reduce repeated information. Based on the above modification principles, the modified second prompt text is obtained: "Please select the course and level you want to study (such as mathematics beginners, English intermediate, English advanced), and other advanced courses are also available for you to choose." Or, for greater conciseness and clarity, it can be further adjusted to: "Please select the course and level (mathematics beginners / English intermediate / English advanced, etc.), and advanced courses are also available." Through this modification process, a second prompt text that is both accurate and concise is obtained, which can better guide users to make choices and learn.

[0121] By adding logical branches or merging existing statements, we can ensure that the secondary prompt text more accurately reflects business needs and user intent. The revised secondary prompt text is more concise and clear, reducing user confusion and misunderstandings and improving the user experience. By merging redundant information or simplifying language, we can reduce the time and resources consumed by the system to process text, thereby improving system efficiency.

[0122] In some embodiments, the method further comprises: obtaining new output content generated by the first large language model based on the second prompt text for the second Query; obtaining a score score output by the second large language model for the new output content; and determining the second prompt text as the final prompt text for the plurality of first Queries when the score score reaches a second threshold.

[0123] In some embodiments, the second threshold is a minimum requirement for the quality of the output content, which is used to determine whether the second prompt text is effective.

[0124] In some embodiments, the second prompt text is input into the first large language model, which generates new output content for the second Query (which can be obtained by converting or refining the first Query) based on the second prompt text. The new output content should be closely related to the second prompt text and reflect the intention of the second Query. The new output content is input into the second large language model (which can be a model specifically designed to evaluate the quality of the output content, such as a deep learning-based text quality evaluation model). The second large language model evaluates the new output content and outputs a score score, which represents the quality of the new output content. If the score score of the new output content reaches or exceeds the second threshold, the second prompt text can be considered effective and can guide the language model to generate high-quality output content.

[0125] Consider an intelligent assistant on an online shopping platform that uses a language model to answer user queries. Users may enter multiple product-related queries (the first query), such as "I want to buy a cost-effective mobile phone" or "Recommend a mobile phone with a good camera." Initially, a prompt (such as "Please tell me the type and features of the product you want to buy") is used to guide the first language model to generate output. However, this prompt may not be specific enough, resulting in inaccurate or relevant output. Therefore, based on user queries and feedback, the prompt is modified and optimized to produce a second prompt (such as "Please tell me that you want to buy a mobile phone and specify your requirements for cost-effectiveness or camera features"). This second prompt is fed into the first language model to generate a new output (such as "Based on your needs, I recommend buying a mobile phone from brand XX; it offers a high cost-effectiveness and excellent camera features"). This new output is then fed into the second language model for evaluation. The second language model outputs a score (e.g., 85), indicating that the new output is of high quality. Because the score of the new output content reaches the second threshold (80 points), the second prompt text is determined as the final prompt text for multiple first queries. This process ensures that the intelligent assistant's answers are more accurate and useful, improving the user's shopping experience.

[0126] By introducing a second language model to evaluate the output, we ensure that the final prompt text guides the language model to generate high-quality output. By setting a threshold and comparing the scores, we can more objectively select the final prompt text, avoiding the influence of subjective judgment.

[0127] The schematic diagram of the architecture of the prompt text generation system based on the large language model is as follows: Figure 2 The prompt text generation system based on the large language model includes three modules: a generalized scoring agent 201, a suggestion generator 202, and an optimizer 203. These three modules work together to form a closed-loop process, automatically iterating and optimizing the target prompt.

[0128] In some embodiments, the universal scorer 201 is configured to incorporate expected response behavior as an evaluation criterion, compare the output of the first language model with the expected response behavior, and output an evaluation score and detailed explanation. This reduces annotation costs and improves evaluation efficiency and flexibility.

[0129] In some embodiments, suggestion generator 202 is configured to generate general, broadly applicable modification suggestions based on low-scoring data in the output of the first language model. Specifically, suggestion generator 202 generates general, broadly applicable modification suggestions by integrating the second query and its corresponding output of the first language model, the expected response behavior, the explanation output by universal scorer 201, and the current prompt. Furthermore, suggestion generator 202 simultaneously considers multiple similar error cases to generate more general modification suggestions.

[0130] In some implementations, optimizer 203 is configured to iteratively modify the current prompt based on the modification suggestions output by suggestion generator 202. Specifically, optimizer 203 considers the combined impact of multiple modification suggestions to ensure that the prompt is optimized in the correct direction. Furthermore, optimizer 203 adheres to specific principles, such as prioritizing adding logical branches or merging existing statements, and maintaining the overall structure of the prompt.

[0131] It should be noted that the universal scorer 201, suggestion generator 202, and optimizer 203 can all be implemented using models. For example, the universal scorer 201 is implemented using the second largest language model, the suggestion generator 202 is implemented using the third largest language model, and the optimizer 203 is implemented using the fourth largest language model. Alternatively, the universal scorer 201, suggestion generator 202, and optimizer 203 are all implemented using the second largest language model. Here, the second largest language model, the third largest language model, and the fourth largest language model are all pre-trained large language models.

[0132] In this way, by integrating scoring, suggestion generation and optimization modules, the labor costs of R&D and business personnel in the prompt optimization link can be significantly reduced, an efficient and low-cost prompt optimization process can be achieved, and it can adapt to the needs of different business scenarios.

[0133] The Prompt automatic optimization solution disclosed in this paper can be widely applied to various application scenarios based on large language models. The following are several specific application examples:

[0134] Intelligent customer service system conversation style tuning:

[0135] Application process: Collect user query samples -> Define expectations -> Use a general scorer to evaluate existing responses -> The suggestion generator proposes improvement suggestions -> The optimizer adjusts prompts -> Iterate optimization.

[0136] Effect: Improve response accuracy and user satisfaction, and reduce manual customer service costs.

[0137] Content Generation Tools:

[0138] Application process: Collect content demand samples -> Define content quality standards -> Evaluate generated content -> Propose improvement suggestions -> Optimize Prompt -> Iterate to improve content quality.

[0139] Results: Improved relevance, creativity, and consistency of content generation.

[0140] Coding assistance tools:

[0141] Application process: Collect programming task samples -> Define code quality standards -> Evaluate generated code -> Propose improvement suggestions -> Optimize prompts -> Iterate to improve code generation quality.

[0142] Effect: Improve the correctness, efficiency and readability of generated code.

[0143] Educational guidance system:

[0144] Application process: Collect student question samples -> Define teaching standards -> Evaluate system answers -> Propose improvement suggestions -> Optimize prompts -> Iterate to improve teaching quality.

[0145] Effect: Improve the pertinence of answers and teaching effectiveness, and adapt to the needs of different students.

[0146] These application examples demonstrate the applicability and potential value of this disclosure in different fields. Through continuous automatic optimization, the performance of large language models in various scenarios can be significantly improved. It is understood that the above application examples are merely illustrative and not restrictive, and this disclosure does not intend to be exhaustive.

[0147] Figure 3 is a flowchart of optimizing prompt text based on a large language model according to an embodiment of the present disclosure, such as Figure 3As shown, the process includes: the first large language model receives a series of queries, which may come from the same user or from different users; the first large language model generates output content for each query based on the first prompt text; the second large language model obtains the output content generated by the first large language model for each query, and uses a universal scorer to generate an evaluation score and explanation for the output content of the first large language model based on the expected response behavior; uses a suggestion generator to determine a second query below the first threshold from multiple first queries, and generates modification suggestions for the first prompt text based on the expected response behavior, the second query and its corresponding output content and explanation; uses an optimizer to modify the first prompt text based on the modification suggestions to obtain an optimized prompt text (i.e., the second prompt text). The first large language model generates new output content for each query based on the second prompt text, and the second large language model gives a score and explanation for the new output content; when the score reaches the second threshold, the current second prompt text is determined as the final prompt text for the multiple first queries.

[0148] The Prompt automatic optimization solution disclosed in this paper significantly helps products based on large language models:

[0149] 1. Reduce labor costs: Through automated processes, developers' manpower input in prompt tuning is significantly reduced.

[0150] 2. Improve optimization efficiency: The systematic optimization process can complete the tuning of a large number of prompts in a short period of time, accelerating product iteration.

[0151] 3. Improve model performance: Through continuous optimization, significantly improve the quality, relevance and consistency of model output.

[0152] 4. Enhanced adaptability: It can quickly adapt to different business scenarios and improve the application flexibility of the product in various fields.

[0153] 5. Reduce evaluation costs: Use expected response behaviors instead of standard answers to significantly reduce data annotation costs and evaluation costs.

[0154] 6. Achieve continuous optimization: Through a self-circulating mechanism, achieve continuous improvement of model performance.

[0155] 7. Improved explainability: The optimization process is transparent and traceable, which helps to understand and further improve model behavior.

[0156] The use of the Prompt automatic optimization solution disclosed in this disclosure will significantly improve the quality and efficiency of products based on large language models.

[0157] The embodiment of the present disclosure provides a prompt text generation device based on a large language model, such as Figure 4 As shown, the prompt text generation device based on the large language model may include: a first acquisition unit 401, used to obtain the output content generated by the first large language model for multiple first queries based on the first prompt text; a second acquisition unit 402, used to obtain the evaluation score and explanation of the output content generated by the second large language model based on the expected response behavior; a first determination unit 403, used to determine the second query from the multiple first queries, the second query is the first query with an evaluation score lower than the first threshold; a first generation unit 404, used to generate a modification suggestion for the first prompt text based on the expected response behavior, the second query and its corresponding output content and explanation; a second generation unit 405, used to modify the first prompt text based on the modification suggestion to obtain the second prompt text.

[0158] In some embodiments, the second acquisition unit 402 includes: a first determination subunit, used to determine the business scenario to which multiple first queries belong; a second determination subunit, used to determine the scoring criteria that match the business scenario; a control subunit, used to control the second largest language model to generate a scoring score for the output content for each first Query according to the scoring criteria; and generating an explanation of the output content for each first Query based on the expected response behavior.

[0159] In some embodiments, multiple first queries correspond to the same expected response behavior, and the expected response behavior is a response standard that is adapted to the business scenario.

[0160] In some embodiments, the first generating unit 404 is used to: when there are multiple second queries, generate modification suggestions for the first prompt text for each second query based on the output content and explanation corresponding to each second query and the expected response behavior.

[0161] In some embodiments, the first generation unit 404 is used to: when there are multiple second queries, classify the multiple second queries according to business characteristics to form multiple batches; wherein the number of second queries contained in each batch is less than the total number of second queries; for each batch, summarize the output content and explanation corresponding to the second queries in the same batch, and combine with the expected response behavior to jointly generate modification suggestions for the first prompt text corresponding to the batch.

[0162] In some embodiments, the second generating unit 405 is configured to: when there are multiple modification suggestions, modify the first prompt text separately for each modification suggestion to obtain multiple candidate second prompt texts; and determine the second prompt text from the multiple candidate second prompt texts.

[0163] In some embodiments, the second generating unit 405 is configured to: when there are multiple modification suggestions, synthesize the specific contents of the multiple modification suggestions, modify the first prompt text, and obtain the second prompt text.

[0164] In some embodiments, the second generation unit 405 is used to: when the first prompt text includes multiple conditions and corresponding execution content, modify the first prompt text according to a preset modification principle, and the modification principle includes at least one of the following: adding logical branches; merging existing statements; wherein the selection of the modification principle is determined according to the specific modification suggestion content and business needs.

[0165] In some embodiments, the first acquisition unit 401 is further configured to acquire new output content generated by the first large language model for the second query based on the second prompt text; the second acquisition unit 402 is further configured to acquire a score for the new output content output by the second large language model. The prompt text generation device based on the large language model may include: a second determination unit 406 ( Figure 4 (not shown) is used to determine the second prompt text as the final prompt text for multiple first queries when the scoring score reaches a second threshold.

[0166] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0167] The prompt text generation device based on the large language model of the embodiment of the present disclosure can realize an efficient and low-cost prompt optimization process and can adapt to the needs of different business scenarios.

[0168] The embodiment of the present disclosure provides a scenario diagram of a method for generating prompt text based on a large language model, such as Figure 5 shown.

[0169] As previously mentioned, the method for generating prompt text based on a large language model provided by the embodiments of the present disclosure is applied to electronic devices. The term "electronic device" is intended to refer to various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The term "electronic device" may also refer to various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0170] Specifically, the electronic device can perform the following operations:

[0171] Obtain output content generated by the first language model for the plurality of first queries based on the first prompt text;

[0172] Obtain the evaluation score and explanation of the output content generated by the second largest language model based on the expected response behavior;

[0173] Determine a second query from the plurality of first queries, the second query being a first query having an evaluation score lower than a first threshold;

[0174] Generate modification suggestions for the first prompt text based on the expected response behavior, the second query and its corresponding output content and explanation;

[0175] The first prompt text is modified based on the modification suggestion to obtain a second prompt text.

[0176] It should be understood that Figure 5 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Figure 5 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0177] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0178] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0179] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0180] like Figure 6As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0181] Various components in device 600 are connected to I / O interface 605, including an input unit 606, such as a keyboard, mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, optical disk, etc.; and a communication unit 609, such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0182] The computing unit 601 can be a variety of general and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the method for generating prompt text based on a large language model. For example, in some embodiments, the method for generating prompt text based on a large language model can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the method for generating prompt text based on a large language model described above can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the prompt text generation method based on a large language model in any other appropriate manner (for example, by means of firmware).

[0183] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0184] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0185] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0186] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0187] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0188] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0189] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0190] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for generating prompt text based on a large language model, comprising: Obtain output content generated by the first language model for multiple first queries based on the first prompt text; Obtaining an evaluation score and an explanation of the output content generated by the second language model based on the expected response behavior; Determine a second query from the plurality of first queries, where the second query is a first query whose evaluation score is lower than a first threshold; Generate modification suggestions for the first prompt text based on the expected response behavior, the second Query and its corresponding output content and explanation; Modify the first prompt text based on the modification suggestion to obtain a second prompt text; The step of obtaining an evaluation score and an explanation of the output content generated by the second language model based on the expected response behavior includes: Determine the business scenarios to which multiple first queries belong; Determine the scoring criteria that match the business scenario; Control the second largest language model to generate a scoring score for the output content for each first Query according to the scoring criteria; and generate an explanation of the output content for each first Query based on the expected response behavior.

2. The method according to claim 1, wherein The multiple first queries correspond to the same expected response behavior, and the expected response behavior is a response standard that is adapted to the business scenario.

3. The method according to claim 1, wherein Generating a modification suggestion for the first prompt text based on the expected response behavior, the second Query, and its corresponding output content and explanation includes: In the case where there are multiple second queries, based on the output content and explanation corresponding to each second query and in combination with the expected response behavior, a modification suggestion for the first prompt text is generated for each second query.

4. The method according to claim 1, wherein Generating a modification suggestion for the first prompt text based on the expected response behavior, the second Query, and its corresponding output content and explanation includes: In the case where there are multiple second queries, the multiple second queries are classified according to business characteristics to form multiple batches; wherein the number of second queries contained in each batch is less than the total number of second queries; For each batch, the output content and explanation corresponding to the second query in the same batch are summarized, and combined with the expected response behavior to jointly generate modification suggestions for the first prompt text corresponding to the batch.

5. The method according to claim 1, wherein The step of modifying the first prompt text based on the modification suggestion to obtain a second prompt text includes: In the case where there are multiple modification suggestions, modify the first prompt text separately for each modification suggestion to obtain multiple candidate second prompt texts; The second prompt text is determined from the multiple candidate second prompt texts.

6. The method according to claim 1, wherein The step of modifying the first prompt text based on the modification suggestion to obtain a second prompt text includes: In the case where there are multiple modification suggestions, the first prompt text is modified based on the specific contents of the multiple modification suggestions to obtain the second prompt text.

7. The method according to claim 1, wherein The step of modifying the first prompt text based on the modification suggestion to obtain a second prompt text includes: In the case where the first prompt text includes multiple conditions and corresponding execution contents, the first prompt text is modified according to a preset modification principle, and the modification principle includes at least one of the following: adding a logical branch; merging existing statements; wherein the selection of the modification principle is determined based on the specific modification suggestion content and business needs.

8. The method according to claim 1, further comprising: Obtain new output content generated by the first language model for the second query based on the second prompt text; Obtaining a scoring score for the new output content output by the second largest language model; When the scoring score reaches a second threshold, the second prompt text is determined as the final prompt text for the multiple first queries.

9. A device for generating prompt text based on a large language model, comprising: A first acquisition unit is configured to acquire output content generated by a first language model for a plurality of first queries based on the first prompt text; a second acquisition unit, configured to acquire an evaluation score and an explanation of the output content generated by a second language model based on the expected response behavior; A first determining unit is configured to determine a second query from a plurality of first queries, where the second query is a first query having an evaluation score lower than a first threshold; A first generating unit, configured to generate a modification suggestion for the first prompt text based on the expected response behavior, the second Query and its corresponding output content and explanation; a second generating unit, configured to modify the first prompt text based on the modification suggestion to obtain a second prompt text; wherein the second obtaining unit includes: A first determining subunit, configured to determine business scenarios to which the plurality of first queries belong; A second determining subunit, configured to determine a scoring criterion matching the business scenario; The control subunit is configured to control the second language model to generate a scoring score for the output content for each first query according to the scoring criteria; and to generate an explanation of the output content for each first query based on the expected response behavior.

10. The device according to claim 9, wherein The multiple first queries correspond to the same expected response behavior, and the expected response behavior is a response standard that is adapted to the business scenario.

11. The device according to claim 9, wherein The first generation unit is configured to: In the case where there are multiple second queries, based on the output content and explanation corresponding to each second query and in combination with the expected response behavior, a modification suggestion for the first prompt text is generated for each second query.

12. The device according to claim 9, wherein The first generation unit is configured to: In the case where there are multiple second queries, the multiple second queries are classified according to business characteristics to form multiple batches; wherein the number of second queries contained in each batch is less than the total number of second queries; For each batch, the output content and explanation corresponding to the second query in the same batch are summarized, and combined with the expected response behavior to jointly generate modification suggestions for the first prompt text corresponding to the batch.

13. The device according to claim 9, wherein The second generating unit is configured to: In the case where there are multiple modification suggestions, modify the first prompt text separately for each modification suggestion to obtain multiple candidate second prompt texts; The second prompt text is determined from the multiple candidate second prompt texts.

14. The device according to claim 9, wherein The second generating unit is configured to: In the case where there are multiple modification suggestions, the first prompt text is modified based on the specific contents of the multiple modification suggestions to obtain the second prompt text.

15. The device according to claim 9, wherein The second generating unit is configured to: In the case where the first prompt text includes multiple conditions and corresponding execution contents, the first prompt text is modified according to a preset modification principle, and the modification principle includes at least one of the following: adding a logical branch; Merge existing statements; wherein the selection of the modification principle is determined according to the specific modification suggestion content and business needs.

16. The device according to claim 9, wherein The first acquisition unit is further configured to acquire new output content generated by the first language model for the second query based on the second prompt text; the second acquisition unit is further configured to acquire a score for the new output content output by the second language model; The device further comprises: The second determining unit is configured to determine the second prompt text as the final prompt text for the multiple first queries when the scoring score reaches a second threshold.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are for causing a computer to execute the method according to any one of claims 1-8.

19. A computer program product comprising a computer program stored on a storage medium, the computer program implementing the method according to any one of claims 1 to 8 when executed by a processor.

Images