Method and device for determining cue word template, electronic equipment and program product

Abstract

The embodiment of the invention relates to a method and device for determining a cue word template, electronic equipment and a program product. The method comprises the step of generating cue words based on data in a data source and a cue word template. A first reasoning result is generated by the baseline model based on the cue word, a second reasoning result is generated by the comparison model based on the cue word, the baseline model is a model used in the service, and the comparison model is used for evaluating the baseline model. And based on the similarity between the first reasoning result and the second reasoning result, determining a score for the cue word. And if the score is greater than or equal to a first threshold value, determining that the cue word template is a target cue word template.

Description

Technical Field

[0001] Embodiments of this specification relate to the field of computers, and more particularly, to a method, apparatus, electronic device, and program product for determining a prompt template. Background Art

[0002] A prompt template is a pre-designed text framework used to guide users to input instructions to an artificial intelligence model more efficiently and accurately, so as to obtain an ideal output result, and it plays a significant role in application scenarios of various generative models. For example, through the prompt template, it can be clearly indicated what specific task the user hopes the model to execute, and the inference result that better meets the user's needs can be output by the model through the prompt template.

[0003] However, after different prompt templates are combined with the given data, the inference results of the model may not meet the user's needs. Therefore, a method for evaluating prompt templates is needed so that a prompt template with higher quality can be selected at the input end of the model to improve the quality of the inference results. Summary of the Invention

[0004] Embodiments of this specification provide a method, apparatus, electronic device, and program product for determining a prompt template.

[0005] According to a first aspect of this specification, a method for determining a prompt template is provided. The method includes generating a prompt based on data in a data source and a prompt template. The method further includes generating a first inference result by a baseline model based on the prompt, and generating a second inference result by a comparison model based on the prompt, where the baseline model indicates the model used in the service, and the comparison model is used to evaluate the baseline model. The method further includes determining a score for the prompt based on the similarity between the first inference result and the second inference result. In addition, the method further includes determining the prompt template as the target prompt template in response to the score being greater than or equal to a first threshold.

[0006] According to a second aspect of this specification, a device for determining a prompt template is provided. The device includes a prompt generation module configured to generate a prompt based on data in a data source and a prompt template. The device further includes an inference result generation module configured to generate a first inference result by a baseline model based on the prompt, and generate a second inference result by a comparison model based on the prompt, where the baseline model indicates the model used in the service, and the comparison model is used to evaluate the baseline model. The device further includes a score determination module configured to determine a score for the prompt based on the similarity between the first inference result and the second inference result. In addition, the device further includes a target prompt template determination module configured to determine the prompt template as the target prompt template in response to the score being greater than or equal to a first threshold.

[0007] According to a third aspect of the present specification, there is provided an electronic device. The electronic device includes at least one processor; and a memory coupled to the at least one processor and having instructions stored thereon, which, when executed by the processor, cause the electronic device to execute the method according to the first aspect above.

[0008] According to a fourth aspect of the present specification, there is provided a computer-readable storage medium having stored thereon a computer program product, the computer program product including machine-executable instructions that, when executed, cause a machine to execute the steps of the method according to the first aspect of the present specification.

[0009] According to a fifth aspect of the present specification, there is provided a computer program product that is tangibly stored on a non-volatile computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to execute the steps of the method according to the first aspect of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] By describing the exemplary embodiments of the present specification in more detail in conjunction with the accompanying drawings, the above and other objects, features, and advantages of the present specification will become more apparent, wherein, in the exemplary embodiments of the present specification, the same reference numerals generally represent the same components.

[0011] Figure 1 A schematic diagram illustrating an example scenario in which the device and / or method according to the embodiments of the present specification may be implemented;

[0012] Figure 2 A schematic flowchart illustrating the method for determining a prompt word template in the embodiments of the present specification;

[0013] Figure 3 A schematic flowchart illustrating the evaluation of prompt words in the embodiments of the present specification;

[0014] Figure 4 A schematic flowchart illustrating the service inference using the prompt word template in the embodiments of the present specification;

[0015] Figure 5 A schematic flowchart illustrating the selection of a prompt word template in the embodiments of the present specification;

[0016] Figure 6 A schematic block diagram illustrating the apparatus for determining a prompt word template provided in the embodiments of the present specification; and

[0017] Figure 7 A schematic block diagram of an example device suitable for implementing the embodiments of the present specification.

[0018] In each of the drawings, the same or corresponding reference numerals denote the same or corresponding parts. Detailed implementation manners

[0019] Embodiments of the present specification will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present specification are shown in the drawings, it should be understood that the present specification can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present specification. It should be understood that the drawings and embodiments of the present specification are only for exemplary purposes and are not used to limit the protection scope of the present specification.

[0020] In the description of the embodiments of the present specification, the term "including" and its like should be understood as an open inclusion, that is, "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. There may also be other explicit and implicit definitions hereinafter.

[0021] As mentioned above, the selection of the prompt template is related to the accuracy of the inference result of the model. In the model input stage, a prompt template needs to be selected for the service scenario. However, due to problems such as the complexity and diversity of the model, the selected prompt template often has the problem that the inference result cannot meet the needs of the service scenario. Therefore, it is necessary to change the prompt template and the sampled data multiple times and input them into the model for calculation, and select the prompt template according to the user's satisfaction with the inference result. This results in a low selection efficiency of the prompt template, and thus it is impossible to quickly obtain an inference result that meets the needs of the service scenario.

[0022] To address at least the above and other potential problems, embodiments of this specification provide a method for determining a prompt template. In this method, prompts can be generated based on data in a data source and a prompt template. A baseline model generates a first inference result based on the prompt, and a comparison model generates a second inference result based on the prompt, where the baseline model is the model used in the service and the comparison model is used to evaluate the baseline model. The score for the prompt can be determined based on the similarity between the first inference result and the second inference result. When the score is greater than or equal to a first threshold, the prompt template is determined as the target prompt template. Through this method, the prompt template and the data in the data source can be combined to generate a prompt, and the baseline model and the comparison model respectively generate inference results regarding the prompt. By comparing the similarity of the inference results output by the two models, the score of the prompt is determined, so as to select the prompt template corresponding to the prompt with a higher score. In this way, the current prompt template can be automatically evaluated through the similarity between the inference results of different models, avoiding manual evaluation of the inference results by the user, improving the efficiency of selecting the prompt template, and thus improving the accuracy of model inference.

[0023] For ease of understanding, first, in combination with Figure 1 describe the scenarios applicable to the embodiments of this specification. In Figure 1 In the shown scenario 100, it includes a prompt template 101, a data source 102, a control device 103, and a model 104. Among them, the prompt template 101 is a structured text framework, which is a standard paradigm for constructing instructions for users to give to artificial intelligence. Exemplarily, the prompt template 101 can have a fixed framework, for example, including topic description, style limitation, length requirement, material reference, and output format.

[0024] In some embodiments, the prompt template 101 can be used to generate e-commerce product descriptions. Structurally, the prompt template 101 can include a task description, limiting conditions, and input data guidelines. By defining in terms of task description, limiting conditions, and input data guidelines, using the prompt template 101 can enable the model to more accurately generate inference results that meet user needs.

[0025] In some embodiments, the data source 102 represents the source or origin of data and can provide raw data materials for artificial intelligence data processing tasks. Exemplarily, the data types in the data source 102 can include offline data, Excel sheet data, information input by users, system logs, real-time stream data, messages, and other data. In some embodiments, when the control device 103 obtains different types of data from the data source 102, it can use extended information according to the type of data to obtain information that can be filled into the prompt template 101. Exemplarily, when the control device 103 obtains data from the system log, it can use the extended information request to obtain only the information related to the prompt template 101 from the system log.

[0026] Extended information refers to the relevant information supplemented and expanded on the basis of the original data, aiming to obtain more complete and comprehensive information. For example, in each record in the database table, in addition to the necessary fields such as the primary key and key attributes, there is also an "extInfo" column to accommodate supplementary details. More detailed information in the database table can be obtained by using the extInfo request.

[0027] In some embodiments, due to different data sources resulting in different data acquisition methods, the control device 103 can obtain the acquisition methods of different data in the extInfo field of the data source 102 according to the type of data. The control device 103 can obtain the information associated with the prompt template by using the extInfo request.

[0028] In some embodiments, the model 104 can process the prompt template 101 and a certain proportion of the data extracted from the data source 102 and generate an inference result. A model is a mathematical and algorithmic abstract representation of complex tasks, systems, or data relationships in the real world, used to simulate, predict, classify, or generate specific data, and is the core element of the entire artificial intelligence technology system. In the field of deep learning, a model is often a large and complex neural network, such as the Transformer model, which is stacked layer by layer by the multi-head attention mechanism, feed-forward neural network, etc. In natural language processing, a large amount of text data flows into the model, and each layer extracts lexical, syntactic, and semantic information respectively, and finally integrates to generate coherent text, accurate translation results, or question-and-answer responses. Deep learning models usually have a large number of parameters to fit complex real-world relationships. These parameters are continuously fine-tuned in large-scale unsupervised text learning and carry language knowledge and semantic understanding rules, enabling the model to handle various text tasks.

[0029] In some embodiments, the control device 103 may select a prompt template, such as the prompt template 101. The control device 103 may extract a certain proportion of data from the data source 102. The types of the extracted data may be one or more of offline data, excel sheet data, user input information, system logs, real-time stream data, messages, etc., which are not limited herein. In some embodiments, the control device 103 may use an extended information request to obtain information on multiple predetermined types of data, where the information is related to the input of the model. Based on the selected prompt template 101 and the extracted certain data, the control device 103 may generate a prompt. In some embodiments, the control device 103 may input the generated prompt into the model 104, and the model 104 may generate a corresponding inference result based on the prompt. Exemplarily, the prompt template 101 indicates to generate an e-commerce product description. The data extracted by the control device 103 from the data source 102 may include, for example, product name, length, field of interest, brand name, and appearance description. Based on the prompt template 101 and the extracted data, the model 104 may generate a product description article related to the product name.

[0030] It should be understood that Figure 1 the illustrated scenario 100 is only an example of this specification and cannot limit this specification. For example, in some embodiments, the control device 103 may be deployed in a computer system, such as a module in the computer system running the model 104. It should also be understood that the control device 103 in the embodiments of this specification is only illustrative, and the control device 103 may be any suitable device and may be implemented in a software and / or hardware manner.

[0031] As described above in connection with Figure 1 the example scenarios in which the devices and / or methods of the embodiments of this specification may be implemented. Next, in connection with Figure 2 the process of determining the prompt template involved in the embodiments of this specification will be described.

[0032] Figure 2 Exemplarily, the method 200 for determining the prompt template in the embodiments of this specification is shown. The method 200 may be executed, for example, by the control device 103 in the environment 100. For ease of explanation, hereinafter, taking the control device 103 as the execution subject as an example, the method 200 will be schematically described. Referring to Figure 2 , the method 200 may include block 202 to block 208.

[0033] In block 202, a prompt is generated based on the data in the data source and the prompt template. In some embodiments, the control device 103 may select a prompt template according to different service scenarios. For example, in an e-commerce scenario, the control device 103 may select a prompt template for product feature extraction. In some embodiments, the control device 103 may extract a certain proportion of data from the data source 102. For example, the control device 103 may extract a certain proportion of product feature data from the data source 102. Based on the selected prompt template and data, the control device 103 may combine the prompt template and data to generate a prompt.

[0034] In block 204, a first inference result is generated by the baseline model based on the prompt, and a second inference result is generated by the comparison model based on the prompt, where the baseline model indicates the model used in the service, and the comparison model is used to evaluate the baseline model. In some embodiments, the baseline model represents the model used in the service scenario. For example, in an e-commerce scenario, the baseline model may be a model for product name inference. The control device 103 may input the generated prompt into the baseline model. The baseline model generates an inference result based on the prompt (hereinafter simply referred to as the first inference result for ease of understanding and explanation).

[0035] In some embodiments, the comparison model represents a model used to evaluate the performance of the baseline model. For example, the comparison model can also be used for product name inference. In some embodiments, the control device 103 may input the generated prompt into the comparison model, and the comparison model generates an inference result based on the prompt (hereinafter simply referred to as the second inference result for ease of understanding and explanation).

[0036] In block 206, a score for the prompt is determined based on the similarity between the first inference result and the second inference result. In some embodiments, the control device 103 may perform a similarity analysis on the first inference result generated by the baseline model and the second inference result generated by the comparison model to determine the similarity between the first inference result and the second inference result, and based on this similarity, determine the score for the prompt. For example, when the first inference result and the second inference result are exactly the same, the control device 103 may determine that the score for the prompt is 100 points. When the similarity between the first inference result and the second inference result is 60%, the control device 103 may determine that the score for the prompt is 60 points. When the similarity between the first inference result and the second inference result is 30%, the control device 103 may determine that the score for the prompt is 30 points.

[0037] In block 208, in response to the score being greater than or equal to the first threshold, the prompt template is determined to be the target prompt template. In some embodiments, when the score of the prompt is greater than or equal to the first threshold, the control device 103 may determine that the inference result generated by the prompt meets the service scenario requirements, and may determine the prompt template corresponding to the prompt as the target prompt template. For example, if the first threshold is set to 60 points, when the score of the prompt is greater than or equal to 60 points, the control device 103 may determine the prompt template corresponding to the prompt as the target prompt template.

[0038] In this way, the current prompt template can be automatically evaluated through the similarity between the inference results of different models, avoiding manual evaluation of the inference results by users and improving the efficiency of selecting the prompt template. Thus, before using the model to generate the inference result, the baseline model and the comparison model can be used to evaluate the quality of the prompt template. In the actual service scenario, based on the prompt template with higher quality, a higher-quality inference result can be generated, improving the inference efficiency and the quality of the inference result.

[0039] The above combination Figure 2 describes the process for determining the prompt template involved in the embodiments of this specification. Next, in combination with Figure 3 describe the embodiments of this specification, Figure 3 FIG. 300 is a schematic flowchart showing the evaluation of prompts in the embodiments of this specification. The flowchart 300 may be executed, for example, by the control device 103 in the environment 100. For ease of explanation, next, taking the control device 103 as the execution subject as an example, the flowchart 300 will be schematically described. Refer to Figure 3 , the flowchart 300 may include block 302 to block 330.

[0040] In block 302, the control device 103 may select a prompt template. In block 304, the control device 103 may select the large model to be evaluated, where the large model to be evaluated may include a baseline model and a comparison model. In block 306, the control device 103 may specify the data source for evaluation. In some embodiments, the data source for evaluation may be the data source 102. In block 308, the control device 103 may determine the type of the data source. In some embodiments, the type of the data source may include the data in the offline table processor in block 310, the data in the excel processor in block 312, the data in other processors in block 314, and the data in the real-time stream processor in block 316.

[0041] In some embodiments, the control device 103 may generate prompt words from different types of data extracted from the data source 102 and the selected prompt word templates, and input the prompt words into the large model to be evaluated in box 318. The large model to be evaluated in box 318 may include large model A in box 320, large model B in box 322, and more large models in box 324. In some embodiments, large model A may be a baseline model, and large model B and more large models may be comparison models. In box 326, the control device 103 may determine an inference result report for the prompt words based on the inference results of multiple large models. In box 328, the control device 103 may evaluate the inference results of multiple large models based on the inference result set report. In box 330, based on the inference results of multiple large models, the control device 103 may determine the evaluation result of the prompt words.

[0042] In some embodiments, in box 304, the control device 103 may select a baseline model (referred to as large model A for ease of understanding and description) according to service requirements. To evaluate the inference performance of the baseline model and the quality of the prompt word templates, the control device 103 may select multiple comparison models. In some embodiments, the control device 103 may select 2 comparison models, where the performance of one comparison model is slightly better than that of the baseline model (referred to as large model B for ease of understanding and description), and the performance of the other comparison model is much better than that of the baseline model (referred to as large model C for ease of understanding and description).

[0043] In some embodiments, the control device 103 may generate a first inference result and a second inference result based on large model A and large model B, and determine the score of the prompt words and update the prompt word template based on the similarity between the first inference result and the second inference result. In some embodiments, when the score of the prompt words is greater than a second threshold, where the second threshold is greater than the first threshold, the prompt word template is updated. Exemplarily, when the second threshold is 98 points, when the score of the prompt words is greater than the second threshold, it indicates that based on the prompt words, the inference results generated by large model A and large model B are very similar. In other words, based on the prompt words, the differences between different models cannot be reflected, so the prompt word template needs to be updated.

[0044] After this comparison ends, the control device 103 can, based on the updated prompt template and the acquired data, use large model A and large model C to respectively generate new inference results. Based on the generated new inference results, determine the score of this updated prompt after this round of comparison. Based on the score of the updated prompt, determine whether the prompt template corresponding to this updated prompt needs to be further updated. In some embodiments, the control device 103 can generate a new updated prompt based on the updated prompt template and the data. The control device 103 can use large model A and large model C to respectively generate inference results, determine the score of the updated prompt based on this inference. When the score is higher than the first threshold, determine that the prompt template corresponding to this updated prompt is the target prompt template. Since the inference cost of a model with excellent performance is relatively high, in this way, it is possible to avoid directly comparing the baseline model with a comparison model that is much better than the baseline model in the first round of comparison, reducing the economic cost of multiple comparisons.

[0045] In some embodiments, the inference result set report can include a table, where the table can include: the input prompt, the names of multiple models, the inference results of multiple models, the comparison results between the baseline model and multiple comparison models, and information such as the inference time consumption, etc. In some embodiments, the control device 103 can evaluate the inference results based on information such as the comparison results between the baseline model and multiple comparison models and the inference time consumption, and determine the score of the prompt based on the information in the inference result set report. In some embodiments, the control device 103 can perform a similarity analysis on the inference results generated by large model A and large model B to determine the score of this prompt. In some embodiments, the control device 103 can also determine the score of the prompt based on the inference time consumption and the similarity analysis. Exemplarily, the control device 103 can extract a larger proportion of data from the data source 102 than the previous round of inference to generate a prompt, and based on this prompt, use large model A and large model B to respectively generate inference results. Based on the similarity of the two inference results and the comparison of the inference time consumption of this inference with that of the previous round of inference, determine the score of the prompt. Through this method, not only can the comparison results be directly obtained from the inference result set report, but also the proportion of the extracted data can be accurately determined, avoiding an increase in the model inference time consumption caused by excessive data extraction.

[0046] In some embodiments, the control device 103 may generate a first inference result and a second inference result based on the large model A and the large model B, determine the score of the prompt word based on the similarity between the first inference result and the second inference result, and update the prompt word template. In some embodiments, when the score of the prompt word is greater than a second threshold, where the second threshold is greater than the first threshold, the prompt word template is updated. In some embodiments, the control device 103 may generate an updated prompt word based on the data in the data source 102 and the updated prompt word template. The control device 103 may generate a new inference result (referred to as the third inference result for ease of understanding and explanation) by the large model A based on the updated prompt word. The control device 103 may generate a new inference result (referred to as the fourth inference result for ease of understanding and explanation) by the large model B based on the updated prompt word. In some embodiments, the control device 103 may determine the score of the updated prompt word based on the similarity between the third inference result and the fourth inference result.

[0047] In some embodiments, after multiple comparisons between the large model A and the large model B and updating the prompt word template, the control device 103 may compare the large model A with the large model C. Exemplarily, the control device 103 may use the updated prompt word template and the data in the data source 102 and input them into the large model A and the large model C, generate two inference results respectively, and determine the similarity score. When the score of the updated prompt word is less than a third threshold, where the third threshold is less than the first threshold, the parameters of the large model A are adjusted. For example, the third threshold may be 30 points. When the score of the updated prompt word is less than 30 points, the parameters of the large model A are adjusted.

[0048] Large model parameter tuning refers to, based on an existing large-scale pre-trained model, adjusting various parameters to optimize the performance of the model in a specific task or scenario, enabling it to better adapt to different application requirements. By comparing the inference results of the baseline model with those of a comparison model whose performance is far superior to the baseline model, the parameters of the baseline model are adjusted to further improve the performance of the baseline model.

[0049] In some embodiments, a prompt is generated based on a selected prompt template and data. The control device 103 may use the large model A with adjusted parameters to generate a new inference result (referred to as the fifth inference result for ease of understanding and explanation) based on the prompt. The control device 103 may use the large model C to generate a new inference result (referred to as the sixth inference result for ease of understanding and explanation) based on the prompt. The control device 103 may determine a score for the prompt based on the similarity between the fifth inference result and the sixth inference result. In some embodiments, when the score of the prompt is greater than a second threshold, the prompt template is updated. Through continuous iteration, the update of the prompt template and the tuning of the baseline model are completed. Through the above method, the prompt template can be continuously optimized and high-quality prompt templates can be selected for inference, improving the quality of model inference.

[0050] It should be understood that the above description of the evaluation of the baseline model and the comparison model is only an example of the embodiments of this specification and should not constitute a limitation on the solutions provided in this specification. For example, the comparison model in the large model for evaluation may include one or more models, not limited to the two comparison models described above.

[0051] The above combination Figure 3 describes the process for determining the prompt template involved in the embodiments of this specification. Next, in combination with Figure 4 describe the embodiments of this specification, Figure 4 FIG. 400 is a schematic flowchart showing service inference using a prompt template in the embodiments of this specification. The flowchart 400 may be executed, for example, by the control device 103 in the environment 100. For ease of explanation, next, taking the control device 103 as the execution subject as an example, the flowchart 400 will be schematically described. Referring to Figure 4 , the flowchart 400 may include block 402 to block 414.

[0052] In block 402, a data source is determined and a certain proportion of data is extracted from the data source. In block 404, a target prompt template is selected from the prompt platform. In block 406, a target prompt is determined based on the extracted certain proportion of data and the target prompt template, and the baseline model is used to generate a target inference result. In block 408, the target inference result is input into a downstream task. For example, it may be block 410 to apply the target inference result to an online service, block 412 to display the target inference result to a user through a page, or block 414 to store the target inference result in a database for use by downstream tasks.

[0053] In some embodiments, in the foregoing block 404, the control device 103 may select a target prompt template from a prompt platform, where the prompt platform includes the scores of prompts evaluated by a baseline model and a comparison model. When the score of a prompt is greater than or equal to a first threshold, the control device 103 may select a prompt template as the target prompt template.

[0054] In some embodiments, in the foregoing block 406, the control device 103 may determine a target prompt based on a certain proportion of the extracted data and the target prompt template, and generate a target inference result using the baseline model with adjusted parameters. Through this method, not only an optimized prompt template is selected, but also the parameters of the baseline model are adjusted, further improving the model inference effect.

[0055] In some embodiments, the control device 103 may analyze the target inference result. For example, it may perform a coverage analysis on the target inference result, where the coverage represents the ratio of the number of obtained inference results to the extracted data. When the coverage is low, the control device 103 may determine whether to increase computing resources, whether to throttle the data stream input to the baseline model, and whether to adjust the prompt template.

[0056] In some embodiments, the control device 103 may perform an accuracy analysis on the target inference result to determine whether the target inference result is consistent with human cognition and avoid the model having "hallucinations". Exemplarily, the control device 103 may extract a certain proportion of the data and compare it with the inference result of the comparison model to determine the accuracy of the inference result. When the accuracy is lower than a threshold, the control device 103 may use a Retrieval-Augmented Generation (RAG) model to improve the accuracy.

[0057] In some embodiments, the control device 103 may perform a user bias analysis on the target inference result to determine whether the target inference result is related to user preferences. Exemplarily, when the bias analysis determines that a user may have an 85% preference to purchase a certain product, compared with a 45% user preference degree, the target inference result will be more in line with the service requirements at the 85% user preference degree.

[0058] In some embodiments, the control device 103 may store the target inference result. When receiving a request to obtain the target inference result, the control device 103 may send the target inference result to other devices for further model training. For example, based on the target inference result, a new model may be further trained. By performing the above steps, based on the updated target prompt template and the baseline model with adjusted parameters, inference results more in line with the needs of downstream tasks can be generated.

[0059] The above combination Figure 4The process of service inference using a prompt template involved in the embodiments of this specification is described. Next, in combination with Figure 5 the embodiments of this specification will be described. Figure 5 FIG. 500 is a schematic flowchart showing the selection of a prompt template in the embodiments of this specification. The flowchart 500 can be executed, for example, by the control device 103 in the environment 100. For ease of explanation, next, taking the control device 103 as the execution subject as an example, the flowchart 500 will be schematically described. Referring to Figure 5 , the flowchart 500 may include block 502 to block 518.

[0060] In block 502, in response to the input of the baseline model, in block 504, the usage mode of the prompt template is determined. In web usage, for example, in block 506, a prompt template is selected on the web side. In block 508, the data source is determined. For example, the data source can be the excel processor in block 510 or the text processor in block 512. In block 514, based on the selected prompt template and data, the inference result is displayed on the web side. In program-side usage, for example, in block 516, a prompt template SDK is selected. In block 518, based on the selected prompt template and the selected data, the output inference result is obtained.

[0061] In some embodiments, in the aforementioned block 516, the prompt template SDK may include an authentication logic, where the authentication logic is a set of rules and processing procedures for verifying and confirming the identity of an entity and the corresponding permissions. By verifying and confirming the access to the prompt template SDK, the security of accessing the prompt template can be improved.

[0062] It should be understood that the above steps of updating the prompt template and adjusting the model parameters are only examples. The number of the baseline model and the comparison model may include one or more, which is not limited herein.

[0063] Through the method of this specification, the data in the prompt template and the data source can be combined to generate a prompt. Based on this prompt, the baseline model and the comparison model respectively generate inference results regarding this prompt. By comparing the inference results output by the two models, the score of the prompt can be determined, so as to select the prompt template corresponding to the prompt with a higher score, thereby avoiding selecting a prompt template with a lower score. In addition, the parameters of the baseline model can also be adjusted to improve the model inference quality.

[0064] Figure 6 FIG. 600 is a schematic diagram of a device for determining a prompt template provided by the embodiments of this specification. As Figure 6As shown, the device 600 may include a prompt word generation module 602 configured to generate prompt words based on the data in the data source and the prompt word template. The device 600 further includes an inference result generation module 604 configured to generate a first inference result by the baseline model based on the prompt word and a second inference result by the comparison model based on the prompt word, where the baseline model indicates the model used in the service and the comparison model is used to evaluate the baseline model. The device 600 further includes a score determination module configured to determine a score for the prompt word based on the similarity between the first inference result and the second inference result. In addition, the device 600 further includes a target prompt word template determination module 606 configured to determine the prompt word template as the target prompt word template in response to the score being greater than or equal to the first threshold.

[0065] In some embodiments, the device 600 further includes: a prompt word template update module configured to update the prompt word template in response to the score being greater than a second threshold, where the second threshold is greater than the first threshold; and a baseline model parameter adjustment module configured to adjust the parameters of the baseline model in response to the score being less than a third threshold, where the third threshold is less than the first threshold.

[0066] In some embodiments, the prompt word template update module further includes: a prompt word update module configured to generate updated prompt words based on the data in the data source and the updated prompt word template; a first inference module configured to generate a third inference result by the baseline model based on the updated prompt word and a fourth inference result by the comparison model based on the updated prompt word; and a first score determination module configured to determine a score for the updated prompt word based on the similarity between the third inference result and the fourth inference result.

[0067] In some embodiments, the baseline model parameter adjustment module further includes: a second inference module configured to generate a fifth inference result by the baseline model with adjusted parameters based on the prompt word and a sixth inference result by the comparison model based on the prompt word; and a second score determination module configured to determine a score for the prompt word based on the similarity between the fifth inference result and the sixth inference result.

[0068] In some embodiments, the device 600 further includes: a third inference module configured to generate an inference result by the baseline model based on the target prompt word template and the data in the data source, where the inference result is used for a downstream task.

[0069] In some embodiments, the third inference module further includes a target prompt word template selection module configured to, in response to an inference request of the baseline model, select a target prompt word template from the prompt word template library in the following two ways: select the target prompt word template from a web page or select the target prompt word template from a prompt word template SDK.

[0070] In some embodiments, the apparatus 600 further includes: a data acquisition module configured to acquire at least one of a plurality of predetermined types of data from a data source, where the plurality of predetermined types includes offline table data, excel data, and real-time stream data. And an information acquisition module configured to acquire information on the plurality of predetermined types of data using an extended information request, where the information is related to the inputs of a baseline model and a comparison model.

[0071] Figure 7 A schematic block diagram of an example device 700 that can be used to implement the embodiments of the present specification is shown. Figure 1 The control device 103 in can be implemented using the device 700. As Figure 7 shown, the device 700 includes a computing unit 701 that can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 702 or computer program instructions loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704. Although not shown in Figure 7 the device 700 may further include a coprocessor.

[0072] Multiple components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disc, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0073] Each of the methods or processes described above can be executed by the computing unit 701. For example, in some embodiments, the method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps or actions of the methods or processes described above can be executed.

[0074] This specification may be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for performing various aspects of the present disclosure.

[0075] A computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example—but not limited to—an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or raised structures in grooves having instructions stored thereon, and any suitable combination of the foregoing. The computer-readable storage medium as used herein is not to be construed as a transitory signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0076] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to respective computing / processing devices, or may be downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0077] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine - related instructions, microcode, firmware instructions, state - setting data, or source code or object code written in any combination of one or more programming languages, including object - oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer - readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand - alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, by using the state information of the computer - readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field - programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer - readable program instructions to implement various aspects of the present disclosure.

[0078] Aspects of the present specification are described herein with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present specification. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer - readable program instructions. The various embodiments in the present specification are described in a progressive manner. The same or similar parts among the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for hardware + program - type embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments.

[0079] These computer - readable program instructions can be provided to a processing unit of a general - purpose computer, a special - purpose computer, or other programmable data - processing apparatus, thereby producing a machine such that when these instructions are executed by the processing unit of the computer or other programmable data - processing apparatus, a device is produced that implements the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer - readable program instructions can also be stored in a computer - readable storage medium, and these instructions cause a computer, a programmable data - processing apparatus, and / or other devices to work in a specific manner. Thus, the computer - readable medium storing the instructions includes a manufactured article that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0080] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, causing a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process such that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0081] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present specification. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by a combination of dedicated hardware and computer instructions.

[0082] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain implementations, multitasking and parallel processing are also possible or may be advantageous.

[0083] The embodiments of the present specification have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art in the technical field without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, the practical application, or the improvement of the technology in the market, or to enable other ordinary skilled persons in the technical field to understand the embodiments disclosed herein.

[0084] The above are only optional embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for determining a prompt template, comprising: Generating a prompt based on data in a data source and a prompt template; Generating a first inference result by a baseline model based on the prompt, and generating a second inference result by a comparison model based on the prompt, wherein the baseline model indicates the model used in the service, and the comparison model is used to evaluate the baseline model; Determining a score for the prompt based on the similarity between the first inference result and the second inference result; And In response to the score being greater than or equal to a first threshold, determining the prompt template as the target prompt template.

2. The method according to claim 1, further comprising: Updating the prompt template in response to the score being greater than a second threshold, wherein the second threshold is greater than the first threshold; Or Adjusting the parameters of the baseline model in response to the score being less than a third threshold, wherein the third threshold is less than the first threshold.

3. The method according to claim 2, wherein updating the prompt template in response to the score being greater than the second threshold comprises: Generating an updated prompt based on data in the data source and the updated prompt template; Generating a third inference result by the baseline model based on the updated prompt, and generating a fourth inference result by the comparison model based on the updated prompt; Determining a score for the updated prompt based on the similarity between the third inference result and the fourth inference result.

4. The method according to claim 2, wherein adjusting the parameters of the baseline model in response to the score being less than the third threshold comprises: Generating a fifth inference result by the baseline model with adjusted parameters based on the prompt, and generating a sixth inference result by the comparison model based on the prompt; Determining a score for the prompt based on the similarity between the fifth inference result and the sixth inference result.

5. The method according to claim 1, further comprising: Generating an inference result by the baseline model based on the target prompt template and data in the data source, wherein the inference result is used for a downstream task.

6. The method according to claim 5, further comprising: In response to an inference request of the baseline model, selecting the target prompt template from a prompt template library in the following two ways: selecting the target prompt template from a web page or selecting the target prompt template from a prompt template SDK.

7. The method according to claim 1, further comprising: Obtaining at least one of a plurality of predetermined types of the data from the data source, wherein the plurality of predetermined types include offline table data, excel data, and real-time stream data; And Using an extended information request to obtain information about the plurality of predetermined types of the data, wherein the information is related to the input of the baseline model and the comparison model.

8. An apparatus for determining a prompt template, comprising: A prompt generation module configured to generate a prompt based on data in a data source and a prompt template; An inference result generation module, configured to generate a first inference result based on the prompt by a baseline model, where the baseline model indicates the model used in the service, and generate a second inference result based on the prompt by a comparison model, where the comparison model indicates the model for evaluating the baseline model; A score determination module, configured to determine a score for the prompt based on the similarity between the first inference result and the second inference result; And A target prompt template determination module, configured to determine the prompt template as the target prompt template in response to the score being greater than or equal to a first threshold.

9. An electronic device, comprising: At least one processor; And A memory, coupled to the at least one processor and having instructions stored thereon, the instructions when executed by the at least one processor cause the device to perform the method according to any one of claims 1 to 7.

10. A computer program product, the computer program product comprising machine-executable instructions, the machine-executable instructions when executed cause the method according to any one of claims 1 to 7 to be implemented.

Images