Model determination method and device, electronic equipment and storage medium

Abstract

The invention provides a model determination method, and relates to the technical field of artificial intelligence, in particular to the technical field of large models, deep learning technologies and big data. According to the specific implementation scheme, according to at least one feature evaluation value corresponding to at least one first target feature tag of a first target sample text, a model evaluation value of a target scene is determined, and the first target sample text is a sample text of the target scene; according to the model evaluation value, a target model is determined from the deep learning models of the multiple different parameter quantities, and the target model is used for processing the problem text of the target scene. The invention further provides a model determination device, electronic equipment and a storage medium.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of large models, deep learning technology and big data technology. More specifically, the present disclosure provides a model determination method, device, electronic device and storage medium. Background Art

[0002] With the development of artificial intelligence technology, the application scenarios of large models are constantly increasing. As the number of parameters of large models increases, the generalization ability of large models is constantly improving. Summary of the invention

[0003] The present disclosure provides a model determination method, apparatus, device and storage medium.

[0004] According to one aspect of the present disclosure, a model determination method is provided, the method comprising: determining a model evaluation value of a target scene based on at least one feature evaluation value corresponding to at least one first target feature label of a first target sample text, wherein the first target sample text is a sample text of the target scene; and determining a target model from a plurality of deep learning models with different parameter amounts based on the model evaluation value, wherein the target model is used to process a problem text of the target scene.

[0005] According to another aspect of the present disclosure, a model determination device is provided, which includes: a first determination module, used to determine a model evaluation value of a target scene based on at least one feature evaluation value corresponding to at least one first target feature label of a first target sample text, wherein the first target sample text is a sample text of the target scene; and a second determination module, used to determine a target model from a plurality of deep learning models with different parameter amounts based on the model evaluation value, wherein the target model is used to process a problem text of the target scene.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method provided according to the present disclosure is implemented.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is a schematic diagram of an exemplary system architecture to which a model determination method and apparatus can be applied according to an embodiment of the present disclosure;

[0012] Figure 2 is a flow chart of a model determination method according to an embodiment of the present disclosure;

[0013] Figure 3 is a schematic diagram of a model determination method according to an embodiment of the present disclosure;

[0014] Figure 4 is a schematic diagram of a sample processing method according to an embodiment of the present disclosure;

[0015] Figure 5 is a schematic diagram of a sample processing method according to an embodiment of the present disclosure;

[0016] Figure 6 is a schematic diagram of multiple feature tags according to an embodiment of the present disclosure;

[0017] Figure 7 is a block diagram of a model determination apparatus according to an embodiment of the present disclosure; and

[0018] Figure 8 is a block diagram of an electronic device to which a model determination method can be applied according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0020] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0021] In the technical solution disclosed in this disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0022] Flagship large models with hundreds of billions of parameters have strong generalization capabilities and a wide range of task adaptability. However, the training and deployment of large models with hundreds of billions of parameters often require massive amounts of data, huge computing resources, and long training cycles. As a result, the application cost of large models with hundreds of billions of parameters is high and difficult to implement. Compared with large models with hundreds of billions of parameters, models with billions or tens of billions of parameters require fewer computing resources and sample data, and have lower application costs. However, models with billions or tens of billions of parameters are difficult to fully utilize the effective information in the data, and the model structure is relatively simple, making it difficult to meet the needs of complex tasks. Their performance may be far inferior to large models with hundreds of billions of parameters.

[0023] In order to reduce application costs, you can try to use a model with smaller parameters to replace a large model. However, the process of judging whether a model with smaller parameters can replace a large model in a specific scenario is a cumbersome process that requires manual intervention. When using a smaller model to replace a large model, some of the generalization capabilities of the large model can be transferred to the smaller model based on technologies such as knowledge distillation or model distillation.

[0024] However, the smaller models obtained based on knowledge distillation and model distillation are often designed and optimized for specific scenarios or task types, lacking comprehensive consideration and universal construction for a variety of different scenarios. Once the scenario changes, the model may need to be readjusted or trained, and cannot flexibly adapt to differences in data distribution, task requirements, and other aspects in different scenarios. In complex and changeable actual application environments, it is difficult to efficiently use the same model architecture and training strategy based on knowledge distillation and model distillation technology to achieve stable performance in different scenarios, limiting the application of distillation technology in a wider range of fields.

[0025] If a large model is compressed, it is easy to lose accuracy. For some text analysis tasks that require high accuracy, over-compressing the model may make the analysis results inaccurate, making it difficult to strike a balance between complexity and accuracy. In addition, when fine-tuning a smaller model, it is often necessary to collect a large number of question-answer pairs for model fine-tuning. Although a large model can generate question-answer pairs, relying entirely on the question-answer pairs provided by the large model for a specified task will make it difficult for the small model to exceed the performance ceiling of the large model. If manually annotated question-answer pairs are used, the fine-tuning cost will be higher.

[0026] Therefore, in order to effectively reduce the cost of using the model under limited resource conditions, the present disclosure provides a model determination method, which will be described below.

[0027] Figure 1is a schematic diagram of an exemplary system architecture to which a model determination method and apparatus can be applied according to an embodiment of the present disclosure. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0028] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0029] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.

[0030] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0031] It should be noted that the model determination method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the model determination device provided in the embodiment of the present disclosure can generally be set in the server 105. The model determination method provided in the embodiment of the present disclosure can generally be executed by the first terminal device 101, the second terminal device 102, and the third terminal device 103. Accordingly, the model determination device provided in the embodiment of the present disclosure can generally be set in the first terminal device 101, the second terminal device 102, and the third terminal device 103. The model determination method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the model determination device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0032] It can be understood that the above describes the system architecture of the present disclosure, and the following will describe the method of the present disclosure.

[0033] Figure 2 is a flowchart of a model determination method according to an embodiment of the present disclosure.

[0034] like Figure 2 As shown, the method 200 may include operations S210 to S220.

[0035] In operation S210, a model evaluation value of a target scene is determined according to at least one feature evaluation value corresponding to at least one first target feature label of a first target sample text.

[0036] In the disclosed embodiment, the first target sample text may be a sample text of a target scene. For example, the target scene may be a scene different from a preset scene. The preset scene may be one or more. A preset scene may be a "role dialogue" scene, which requires the model to act as a role and dialogue with the user. The first target sample text may be a question text provided by the user, which may have a real reply text or may not have a reply text.

[0037] In the embodiments of the present disclosure, the first target feature label can be determined in various ways. The first target feature label can correspond to a feature evaluation value. For example, multiple first target feature labels can be determined from one or more preset feature labels of each of multiple preset sample texts. The preset sample text can be a preset common text. The preset feature label can correspond to a feature evaluation value. If a preset feature label is determined as the first target feature label from multiple preset feature labels, the feature evaluation value corresponding to the preset feature label can be used as the feature evaluation value corresponding to the first target feature label. For another example, one or more first target feature labels can be determined based on the semantics of the first target sample text, and the semantic evaluation value of the first target feature label can be determined as the feature evaluation value. The semantic evaluation value can indicate the accuracy of processing the first target sample data using a smaller model.

[0038] In the embodiments of the present disclosure, various operations are performed based on one or more feature evaluation values ​​to determine a model evaluation value. For example, an average value of multiple feature evaluation values ​​can be determined as the model evaluation value. For another example, the sum of multiple feature evaluation values ​​can also be determined as the model evaluation value.

[0039] In operation S220, a target model is determined from among a plurality of deep learning models with different parameter amounts according to the model evaluation values.

[0040] In the disclosed embodiments, the deep learning models with different parameter amounts may include a large model and a smaller model. For example, the large model may be a model with hundreds of billions of parameters. The smaller model may be a model with tens of billions or billions of parameters.

[0041] In the disclosed embodiment, the model evaluation value may represent the performance of the smaller model in the target scene. If the model evaluation value represents that the performance of the smaller model in the target scene meets the needs of the user, the smaller model may be used as the target model.

[0042] In the disclosed embodiment, the target model is used to process the question text of the target scenario. For example, the question text can be input into the target model to obtain the reply text.

[0043] Through the disclosed embodiments, the characteristic labels of the target sample text can be used to determine the evaluation value of the model. The characteristics of the target sample text can be effectively utilized to accurately determine the target model suitable for the target scene, which can create favorable conditions for reducing the cost of model application while ensuring the accuracy of text processing.

[0044] It can be understood that the above is a description of the method disclosed in the present invention. Figure 3 The method of the present disclosure is further described.

[0045] Figure 3is a schematic diagram of a model determination method according to an embodiment of the present disclosure.

[0046] like Figure 3 As shown, the model determination method can be performed by the feature mining module 31, the data management module 32 and the model routing module 33. For example, the feature mining module 31 can determine one or more first target feature tags of the first target sample text based on the annotated large model TAG30. The feature mining module 31 can also determine the model evaluation value, and the feature mining module 31 will be described below.

[0047] In some embodiments, in some implementations of the above operation S210, determining the model evaluation value of the target scene according to at least one feature evaluation value corresponding to at least one first target feature tag of the first target sample text includes: determining at least one first target feature tag of the first target sample text using the large model according to the annotation prompt data. Through the embodiments of the present disclosure, the feature tag of the first target sample text can be determined quickly and accurately using the large model, which helps to efficiently determine the model evaluation value, reduce the annotation cost of the text, and lower the threshold for model application.

[0048] In the embodiment of the present disclosure, the annotation prompt data may instruct the large model to perform multiple annotation operations. For example, the annotation prompt data may instruct the annotation large model TAG30 to perform multiple annotation operations.

[0049] like Figure 3 As shown, the user can provide one or more first target sample data of the target scene to the annotation model TAG30. According to the annotation prompt data and each first target sample data, the annotation model TAG30 can determine at least one first target feature tag. Taking the at least one first target feature tag as an example, the multiple first target feature tags may include: Figure 3 The first target feature label lable_h31, the first target feature label lable_h32 and the first target feature label lable_h33 are shown.

[0050] In the embodiment of the present disclosure, according to the preset template to be annotated, the first target sample text is used as the input sample text. The preset template to be annotated may include an input text field. The first target sample text may be used as the value of the input text field.

[0051] In an embodiment of the present disclosure, the target text to be annotated is input into the large model so that the large model outputs at least one target label data of the first target sample text according to a preset annotation result template. The target label data includes a plurality of annotation result fields consistent with the preset annotation result template. The plurality of annotation result fields include a feature label field and a label explanation field. The value of the feature label field of the target label data is the first target feature label. The label explanation field of the target label data can represent the meaning of the first target feature label. For example, if the first target feature label is "text analysis", the semantics of the first target feature label can be "the task involves analyzing a given text to identify and extract keywords according to specific criteria to prove the need for text analysis functions". It can be understood that the value of the feature label field can be text, and the value of the label explanation field can also be text. Through the embodiment of the present disclosure, based on the preset template to be annotated and the preset annotation result template, the accuracy of the feature labels determined by the large model can be effectively improved to determine an accurate model evaluation value.

[0052] In the embodiment of the present disclosure, the first target feature tag may be determined from a plurality of preset feature tags. The preset feature tag may be determined using a large model based on the annotation prompt data. Figure 3 As shown, before applying one of the multiple deep learning models with different parameter amounts to the target scene, the annotation model TAG30 can be used to determine one or more feature labels of each of the multiple sample texts to be processed according to the annotation prompt data. Next, the sample text to be processed can be used as the preset sample text, and the feature label of the sample text to be processed can be used as the preset feature label.

[0053] In some embodiments, a preset feature label corresponds to a feature evaluation value. The feature evaluation value of the feature label can be determined in various ways. For example, when the preset feature label of the preset sample text is determined by the annotation large model, the initial evaluation value of the preset feature label of the preset sample text can be determined. There can be multiple preset sample texts with the preset feature label. Thus, based on the multiple initial evaluation values ​​of the preset feature label, the feature evaluation value of the preset feature label can be determined. It can be understood that various operations (such as averaging) can be performed based on the multiple initial evaluation values ​​of the preset feature label to obtain the feature evaluation value. Multiple preset sample texts and one or more preset feature labels of each of the multiple preset sample texts can be stored in the data warehouse db30.

[0054] In the disclosed embodiment, the large model can determine one or more preset feature labels of multiple preset feature labels as one or more first target feature labels. For example, if a preset feature label is used as the first target feature label label_h31, the feature evaluation value of the preset feature label can be used as the feature evaluation value of the first target feature label label_h31.

[0055] In some embodiments, in some implementations of the above operation S210, determining the model evaluation value of the target scene according to at least one first target feature tag of the first target sample text includes: determining the model evaluation value of the target scene according to at least one feature evaluation value of at least one first target feature tag. If there are multiple first target sample texts, for each first target sample text, the sample evaluation index value of the first target sample text can be determined according to at least one feature evaluation value. For example, if the first target sample text has n first target feature tags, the sample evaluation index value can be determined by the following formula:

[0056] (Formula 1)

[0057] n may be an integer greater than or equal to 1. i may be an integer greater than or equal to 1 and less than or equal to n. label_i The feature evaluation value of the i-th first target feature label can be C. query It can be the sample evaluation index value of the first target sample text. It can be understood that the n values ​​of different first target sample texts can be the same or different.

[0058] Next, an addition operation is performed based on the multiple sample evaluation index values ​​of each of the multiple first target sample texts to determine the model evaluation value of the target scene. It can be understood that various operations can be performed based on the multiple sample evaluation index values ​​to determine the model evaluation value. The various operations can include averaging, weighted summing, etc.

[0059] It can be understood that some methods for determining the model evaluation value are described above, and some methods for determining the target model will be described below.

[0060] like Figure 3 As shown, the model routing module 33 can determine the target model from multiple deep learning models with different parameter amounts based on the model evaluation value, which will be explained below.

[0061] In some embodiments, a plurality of deep learning models with different parameter amounts may include a first deep learning model and a second deep learning model. The parameter amount of the first deep learning model may be less than the parameter amount of the second deep learning model. The first deep learning model may be a smaller model with the above-mentioned parameter amount of several billion. The second deep learning model may be a large model with the above-mentioned parameter amount of hundreds of billions. It is understood that the parameter amount of the first deep learning model is less than that of the second deep learning model.

[0062] In an embodiment of the present disclosure, in some implementations of the above operation S220, determining a target model from a plurality of deep learning models with different parameter amounts according to a model evaluation value includes: in response to determining that the model evaluation value is greater than or equal to a preset evaluation threshold, determining a first deep learning model as a target model. Through an embodiment of the present disclosure, a model evaluation value can represent the performance of a smaller model in a target scenario. According to the model evaluation value and the preset evaluation threshold, it can be quickly determined whether the performance of the smaller model meets the requirements of the target scenario. In the case where the performance of the smaller model meets the requirements, the smaller model can be used to process the problem text of the target scenario to effectively reduce computing resource overhead and storage resource overhead.

[0063] In the embodiment of the present disclosure, at least one training sample data may be used to fine-tune the target model to obtain a fine-tuned target model. The at least one training sample data may include at least one of a first target sample text and a second target sample text. The second target sample may be determined from a plurality of preset sample texts according to at least one first target feature tag. Figure 3As shown, when the model evaluation value is greater than or equal to the preset evaluation threshold, multiple preset feature labels consistent with the first target feature label lable_h31, the first target feature label lable_h32, and the first target feature label lable_h33 can be determined from the data warehouse db30 of the data management module 32. The preset sample text having at least one of the multiple preset feature labels is used as the second target sample text. Operation S331 can be performed according to the first target sample text and the second target sample text to fine-tune the first deep learning model. The fine-tuned first deep learning model is obtained. Next, operation S332 can be performed to perform reasoning using the fine-tuned first deep learning model to process the question text of the target scene and obtain the reply text. It can be understood that the first target sample text can have a real reply text. After the first target sample text is input into the first deep learning model, the first sample output result can be obtained. According to the difference between the first sample output result and the real reply text, the first deep learning model can be fine-tuned once. The second target sample text can also have a real reply text. It can also be fine-tuned based on the second target sample text. Thus, after fine-tuning the first deep learning model multiple times using the first target sample text and the second target sample text, a fine-tuned first deep learning model can be obtained. Through the disclosed embodiments, when the first deep learning model with a smaller number of parameters is used as the target model, the feature labels of the first target sample text can be used to determine one or more similar second target sample texts from multiple preset sample texts, which can effectively reduce the cost and difficulty of users providing samples of target scenes, and can effectively fine-tune the first deep learning model using relatively abundant samples to fully improve the efficiency and accuracy of the target model in processing the problem text of the target scene.

[0064] It can be understood that the above description is based on the example that the model evaluation value is greater than or equal to the preset evaluation threshold, but the present disclosure is not limited thereto, and the following description will be based on the example that the model evaluation value is less than the preset evaluation value.

[0065] In an embodiment of the present disclosure, in response to determining that the model evaluation value is less than a preset evaluation threshold, the second deep learning model is determined as the target model. Figure 3 As shown, when the model evaluation value is less than the preset evaluation threshold, operation S333 can be performed to use the second deep learning model for reasoning to process the problem text of the target scene. It can be understood that the second deep learning model has a large number of parameters and strong performance, but the cost is high. In order to reduce costs, the second deep learning model can be fine-tuned without using the first target sample text. It can also be understood that in order to fully improve the accuracy of processing the problem text of the target scene, the second deep learning model can also be fine-tuned using the first target sample text and the second target sample text.

[0066] It can be understood that the above describes the method of the present invention, and the following will describe the annotation prompt data of the present invention.

[0067] In some embodiments, the annotation prompt data includes at least one of a plurality of annotation prompt texts, and the annotation prompt text may instruct the large model to perform an annotation operation corresponding to the annotation prompt text. For example, the large model may be the above-mentioned annotation large model TAG30.

[0068] In an embodiment of the present disclosure, the multiple annotation prompt texts include annotation configuration prompt text, annotation type prompt text, first annotation restriction prompt text, second annotation restriction prompt text and annotation order prompt text.

[0069] In the disclosed embodiment, the annotation prompt text includes at least one of a reference information subtext, an identification subtext, and a core subtext. The reference information subtext can provide reference information for the large model, and the core subtext can include a core word and a core tag. The core word can indicate an operation that the large model needs to perform or does not need to perform. For example, the core subtext can be "**need**", including a core tag "**" and a core word "need".

[0070] In the disclosed embodiment, the annotation configuration prompt text may instruct the large model to determine the feature tags of the input sample text according to the keywords of the input sample text. For example, the annotation configuration prompt text may be "You are a text keyword extractor, and you are very good at extracting keywords that readers need to focus on during the reading process. The extracted keywords need to be evenly distributed in the text and not concentrated in a few sentences. [Keyword extraction rules]". The annotation configuration prompt text may include the identification subtext "[Keyword extraction rules]".

[0071] In the embodiment of the present disclosure, the annotation type prompt text is used to provide the definition of different types of keywords. For example, the annotation type prompt text can be:

[0072] “**Need** to be extracted keyword types:

[0073] + Adjectives: words that can stir up the reader's emotions (e.g., "It's really addictive"), words that can highlight the selling point or characteristics of something (e.g., "sharp without hurting your hands"), words that can reflect contrast (e.g., "affordable price" and "good quality")

[0074] + Quantifiers: words related to the amount of data (such as "90%"), words related to money (such as "10 yuan")

[0075] + People: words related to the target audience (e.g., "students"), names of celebrities (e.g., "Zhang San")

[0076] + Proper nouns: brand or product names, literary and film and television work titles, and other proper nouns, professional terms, and famous quotes

[0077] + Time: words related to time (e.g., "5 minutes", "the sooner the better")

[0078] + Information-related vocabulary: Keywords that reflect the main idea of ​​the text, summary words (such as "two stages", "5 things")

[0079] + Other keywords that need to be focused on. The annotation type prompt text can instruct the annotation model to extract keywords of the type such as adjectives and proper nouns. The tag type prompt text also includes a reference information subtext "(such as: "It's really too addictive")". The reference information subtext can include a preset reference character "such as". In the case of including the preset reference character, the subtext of the prompt text can be used as a reference information subtext.

[0080] In the disclosed embodiment, the annotation prompt text may also include a segmentation mark. The text between two segmentation marks may be a type of keyword and the definition of the keyword of that type. For example, the segmentation mark may be "+". The definition of "adjective" may be "a word that can express the reader's emotions, a word that can highlight the selling point or characteristics of a thing".

[0081] In the disclosed embodiment, the first annotation restriction prompt text may indicate keywords that the large model does not need to extract from the input sample text. For example, the first annotation restriction prompt text may be "Attention!! Modal particles, ordinary names, and words that appear repeatedly in the original text are **not required** to be extracted". The first annotation restriction prompt text may include the core subtext "**not required**". The core subtext includes the core word "not required".

[0082] In the disclosed embodiment, the second annotation restriction prompt text is used to instruct the large model to determine keywords from multiple initial words in the input sample text. For example, the second annotation restriction prompt text may be "The extracted keywords must be a single word contained in the text, and must not be a sentence; when extracting keywords, do not fabricate keywords or splice keywords based on semantics."

[0083] In the disclosed embodiment, the annotation order prompt text is used to instruct the large model to output the keywords according to the text position corresponding to the keywords, and the text position is the position corresponding to the keywords in the input sample text. For example, the annotation order prompt text can be "Please extract the keywords that need to be emphasized in the order in which the keywords appear in the text" according to the order in which they appear in the text. Through the disclosed embodiment, the use of multiple annotation prompt texts can enable the large model to accurately determine the feature labels of the sample text, which helps to efficiently perform subsequent processing.

[0084] It can be understood that the above description is about the annotation prompt data of the present disclosure, and the following description is about the preset template to be annotated and the preset annotation result template of the present disclosure.

[0085] In the embodiment of the present disclosure, the preset template to be annotated includes an input text field. For example, the preset annotation template can be "[text]

[0086] {{

[0087] "instruction": "{question}",

[0088] }}".

[0089] "instruction" can be an input text field. "question" can represent the value of the input text field, or can be an input sample text. The input sample text can be "[text]\\n\\"The God of Wealth is coming oh oh oh\\"\\n\\nPlease strictly follow the [keyword extraction rules] to extract the keywords in [text], please output the keywords:".

[0090] In the embodiment of the present disclosure, the preset annotation result template may include multiple annotation result fields. The multiple annotation result fields may include a feature tag field and a tag explanation field. For example, the preset annotation result template may be:

[0091] "[Output format][

[0092] {

[0093] "tag": str,

[0094] "explanation": str

[0095] },

[0096] {

[0097] "tag": str,

[0098] "explanation": str

[0099] }

[0100] ]”.

[0101] "tag" can be a feature tag field. str indicates that the value type of the annotation result field is a string. "explanation" can be a tag explanation field. The annotation large model can output multiple target tag data of the above-mentioned input sample text according to the preset tag result template. The multiple target tag data may include [{"tag": "Keyword extraction","explanation": "This instruction focuses on extracting specific types of keywords from a given text according to a set of defined rules. It involves identifying adjectives, quantifiers, names, proper nouns, time-related words, information-related vocabulary and other important keywords in the text."} and {"tag": "Text analysis", "explanation": "This task involves analyzing a given text to identify and extract keywords according to specific criteria, thereby demonstrating the need for text analysis functions."}. It can be understood that based on the preset tag result template, the target tag data output by the annotation large model can be more than 2. It can be understood that the first target sample text and the preset sample text above can be used as the values ​​of the input text field respectively.

[0102] It can be understood that the above description uses the example of determining the feature evaluation value based on the initial evaluation value to illustrate the present disclosure, but the present disclosure is not limited thereto, and the feature evaluation value can also be determined based on the evaluation model. The method of determining the feature evaluation value will be further described below.

[0103] In some embodiments, a preset feature label corresponds to a feature evaluation value, and the feature evaluation value is determined based on the sample evaluation results of each of the multiple preset sample texts with the preset feature label. The sample evaluation results of each of the multiple preset sample texts are determined based on the multiple model output results of each of the multiple preset sample texts, and the multiple model output results of the preset sample texts are respectively output by multiple deep learning models with different parameter amounts. Figure 4 Provide explanation.

[0104] Figure 4 is a schematic diagram of a sample processing method according to an embodiment of the present disclosure.

[0105] like Figure 4 As shown, method 400 may be performed before method 200. Method 400 may include operations S401 to S403.

[0106] In operation S401, sample evaluation results of each of the plurality of sample texts to be processed are determined according to the plurality of model output results of each of the plurality of sample texts to be processed.

[0107] In the disclosed embodiment, the multiple model output results of the sample text to be processed are respectively output by multiple deep learning models with different parameter amounts. The multiple deep learning models with different parameter amounts may include a large model and a smaller model. For example, the large model may be a model with hundreds of billions of parameters. The smaller model may be a model with tens of billions or billions of parameters.

[0108] In the disclosed embodiment, the plurality of sample texts to be processed may be natural language texts from one or more scenes. For example, the plurality of scenes may include scenes such as "junior high school composition writing" and "introduction assistant".

[0109] In an embodiment of the present disclosure, a sample evaluation result may indicate: a model that satisfies a preset condition among multiple deep learning models with different parameter amounts. The preset condition may be: the evaluation index value of the model output result is a target evaluation index value among multiple model output results, and the resource overhead is the minimum value of at least one resource overhead corresponding to the target evaluation index value. At least one resource overhead may be the resource overhead of at least one deep learning model corresponding to the target evaluation index value. The target evaluation index value may be the optimal evaluation index value. It is understood that if the evaluation index is accuracy, the target evaluation index value may be the maximum evaluation index value. If the evaluation index is randomness, the target evaluation index value may be the minimum evaluation index value.

[0110] In operation S402, at least one feature tag of each of a plurality of sample texts to be processed is determined.

[0111] In the embodiments of the present disclosure, one or more feature tags of the sample to be processed may be determined in various ways. For example, for a sample text to be processed in the above-mentioned "introduction assistant" scenario, the feature tag may be "text analysis".

[0112] In operation S403, a feature evaluation value of a feature label is determined according to respective sample evaluation results of a plurality of samples to be processed having the same feature label.

[0113] In the disclosed embodiment, the proportion of models with smaller parameter amounts that meet the preset conditions can be determined based on the sample evaluation results of each of the multiple sample texts to be processed with the same feature label. The feature evaluation value can be determined based on the proportion. For example, the proportion is used as the feature evaluation value.

[0114] Through the embodiments of the present disclosure, the feature evaluation value of the feature label of the sample text to be processed can be determined based on the sample evaluation result of the sample text to be processed. Therefore, sample texts with the same features can be mined from the sample texts to be processed in different scenes, and a model suitable for processing the sample text with the feature label can also be determined, which helps to improve the text processing effect and effectively reduce the processing cost. After receiving the text of a new scene, the model for processing the text of the new scene can be quickly determined to achieve efficient text processing at a low cost.

[0115] It can be understood that after the sample text to be processed is processed by the above method 400, the sample text to be processed can be used as a preset sample text. The feature label of the sample text to be processed can be used as a preset feature label of the preset sample text.

[0116] It can be understood that the above describes the sample processing method of the present disclosure, and the following describes the sample text to be processed of the present disclosure.

[0117] Figure 5 is a schematic diagram of a sample processing method according to an embodiment of the present disclosure.

[0118] In some embodiments, data sets commonly used by users in different fields may be collected. Figure 5 As shown, data sets ds50 from different fields can be collected. By cleaning and removing duplicates from the data set ds50, multiple sample texts to be processed can be obtained.

[0119] In some embodiments, multiple sample texts to be processed can be divided into multiple different domain data sets according to the domain of the data set. A domain data set corresponds to one domain. Multiple domains include: "Creation and Continuation", "Legal Moral Security", "Rewriting and Error Correction", "Role Dialogue", "Structured Data Question and Answer", "Mathematics and Logic", "Synonymous Paraphrase and Sample Generation", "Information Extraction", "Reading Comprehension", "Abstract", etc.

[0120] In some embodiments, the sample processing method may include: clustering multiple sample texts to be processed to obtain multiple sample sets to be processed. For example, a word vector conversion operation may be performed on multiple sample texts to be processed to map the sample texts to be processed to a low-dimensional vector space to obtain multiple sample vectors to be processed. Multiple similarities between multiple sample vectors to be processed may be calculated. Multiple sample vectors to be processed with higher similarities are taken as a sample set to be processed. Thus, multiple sample sets to be processed may be obtained. The sample set to be processed may correspond to a scene. Multiple scenes may include "junior high school composition writing", "question analysis", "paper review", "argumentation and argument judgment", etc. Next, operation S5001 may be performed.

[0121] In operation S5001, a plurality of to-be-processed sample sets are used to train a plurality of models with smaller parameter amounts, respectively, to obtain a plurality of trained smaller models. The number of parameters of the smaller models may be, for example, several billion.

[0122] It can be understood that the above describes the sample text to be processed, and some methods for determining the sample evaluation results will be described below.

[0123] In operation S501 , sample evaluation results of each of the plurality of sample texts to be processed are determined according to the plurality of model output results of each of the plurality of sample texts to be processed.

[0124] In the disclosed embodiment, according to the evaluation prompt data, a large model is used to respectively determine a plurality of evaluation index values ​​of a plurality of model output results of the sample text to be processed. According to the plurality of evaluation index values, a sample evaluation result of the sample text to be processed is determined.

[0125] In the embodiment of the present disclosure, the evaluation index value can be determined by using the evaluation large model. It can be understood that the evaluation large model can be the same model as the annotation large model, or can be a different model, and the present disclosure does not limit this.

[0126] In the embodiment of the present disclosure, the evaluation prompt data text includes at least one of a plurality of evaluation prompt texts, and the evaluation prompt text is used to instruct the evaluation large model to evaluate the model output result according to the evaluation method corresponding to the evaluation prompt text.

[0127] In an embodiment of the present disclosure, multiple evaluation prompt texts include evaluation configuration prompt text, accuracy evaluation prompt text, instruction follow prompt text, directness prompt text, fluency evaluation prompt text, compliance evaluation prompt text, consistency evaluation prompt text and evaluation result prompt text.

[0128] In the disclosed embodiment, the evaluation configuration prompt text is used to indicate the evaluation index value of the evaluation big model to determine the model output result. For example, the evaluation configuration prompt text may be "Please act as an impartial judge and evaluate the quality of the answers provided by the AI ​​assistant to the user questions shown below. Your evaluation should consider the following factors:". The "Your evaluation should consider the following factors:" in the evaluation configuration prompt text may instruct the big model to be evaluated according to at least one of the accuracy evaluation prompt text, instruction follow prompt text, directness prompt text, fluency evaluation prompt text, compliance evaluation prompt text, consistency evaluation prompt text and evaluation result prompt text.

[0129] In the embodiment of the present disclosure, the accuracy assessment prompt text is used to indicate the accuracy of the model output result determined by the large model. For example, the accuracy assessment prompt text may be "Accuracy: whether the answer is factually correct.".

[0130] In the disclosed embodiment, the instruction compliance evaluation prompt text is used to indicate whether the evaluation model determines whether the model output result is generated for the sample text to be processed. For example, the instruction compliance prompt text may be "Instruction compliance: whether the answer strictly follows the requirements of the user's question, such as whether all the required information is provided, and the answer should be output in a given sample format. When encountering a choice or classification question, the answer should be output directly without additional explanation."

[0131] In the disclosed embodiment, the directness evaluation prompt text is used to indicate whether the evaluation model determines whether the model output result includes information irrelevant to the sample text to be processed. For example, the directness evaluation prompt text may be "Directness: answer the question directly and positively, without other information and redundant information.".

[0132] In the disclosed embodiment, the fluency evaluation prompt text is used to indicate whether the evaluation model determines whether the model output result meets the preset grammatical conditions. For example, the fluency evaluation prompt text may be "Semantic fluency: whether the answer is clear and fluent, without grammatical errors, incoherence or typos.".

[0133] In the disclosed embodiment, the compliance assessment prompt text is used to indicate whether the evaluation model determines whether the model output result complies with the preset rules. For example, the compliance assessment prompt text may be "Legal ethics and safety: When it comes to sensitive topics, answer whether it complies with legal ethics."

[0134] In the disclosed embodiment, the consistency assessment prompt text is used to indicate whether the model output result is consistent with the reference information in the sample text to be processed. For example, the consistency assessment prompt text may be "Consistency: When reference information is provided in the question, the answer needs to strictly comply with the reference information content and cannot fabricate additional information."

[0135] In the disclosed embodiment, the evaluation result prompt text is used to indicate the numerical range of the evaluation index value. For example, the evaluation result prompt text may be "Please help me evaluate the quality of the answers of the two AI assistants and give corresponding scores from 0 to 10". Through the disclosed embodiment, the use of multiple evaluation prompt texts can enable a large model to accurately determine the evaluation of multiple model output results, which helps to efficiently determine the sample evaluation results.

[0136] It can be understood that the above text describes the evaluation prompt text of the present disclosure, and some methods of determining the evaluation index value of the present disclosure will be described below.

[0137] In the embodiment of the present disclosure, according to the preset template to be evaluated, the sample text to be processed and the output results of multiple models are spliced ​​into the text to be evaluated.

[0138] In the embodiment of the present disclosure, the preset template to be evaluated includes a question field and multiple model output result fields. For example, the preset annotation template can be "[question]

[0139] {

[0140] \"INSTR\": \"{{question_v}}\",

[0141] }

[0142] [Assistant's answer] [

[0144] {

[0145] \"model\": \"model_1\",

[0146] \"answer\": \"{{answerA}}\"

[0147] },

[0148] {

[0149] \"model\": \"model_2\",

[0150] \"answer\": \"{{answerB}}\"

[0151] }

[0152] ]” “INSTR” can be a question field. question_v can represent the value of the question field, which can be the above-mentioned sample text to be processed. “model” can be a model identification field. The model identification field can indicate one of multiple deep learning models with different parameter amounts. “model_1” can represent a first deep learning model. “model_2” can represent a second deep learning model.

[0153] "answer" can be a model output result field. The value of the model output result field can represent one of multiple model output results. "answerA" can represent a model output result. "answerB" can represent a model output result different from "answerB". In the case where "answerA" represents the first model output result output by the first deep learning model, "answerB" can represent the second output result output by the second deep learning model.

[0154] In an embodiment of the present disclosure, based on the evaluation prompt data, using the big model to respectively determine multiple evaluation index values ​​of multiple model output results of the sample text to be processed includes: inputting the text to be evaluated into the big model so that the big model outputs the evaluation data of the sample text to be processed according to a preset evaluation result template.

[0155] In the embodiment of the present disclosure, the evaluation data includes a plurality of evaluation result fields that are consistent with a preset evaluation result template, and the values ​​of the evaluation result fields are evaluation index values.

[0156] For example, the preset evaluation result template can be: "[output format]

[0157] {

[0158] "model1_score": "",

[0159] "model2_score": ""

[0160] }

[0161] Please give an analysis first, and then give the scores of the two models according to the format.

[0162] [Output]".

[0163] "model_1_score" and "model_2_score" may be two evaluation result fields. The value of the evaluation result field may be an evaluation index value. Through the disclosed embodiment, based on the preset template to be evaluated and the preset evaluation result template, the accuracy of the evaluation index value determined by the large model may be effectively improved to determine an accurate sample evaluation result.

[0164] It can be understood that some methods for determining evaluation index values ​​are described above, and some methods for determining sample evaluation results will be described below.

[0165] In an embodiment of the present disclosure, a plurality of deep learning models with different parameter amounts may include a first deep learning model and a second deep learning model. The parameter amount of the first deep learning model may be less than the parameter amount of the second deep learning model. The plurality of model output results include a first model output result output by the first deep learning model and a second model output result output by the second deep learning model. The plurality of evaluation index values ​​include a first evaluation index value of the first model output result and a second evaluation index value of the second model output result.

[0166] In some embodiments, the sample evaluation result may be a Good Same Bad (GSB) result.

[0167] In some embodiments, according to multiple evaluation index values, determining the sample evaluation result of the sample text to be processed includes: in response to determining that the first evaluation index value is greater than or equal to the second evaluation index value, determining that the sample evaluation result of the sample text to be processed is the first sample evaluation result. The first sample evaluation result may indicate that when processing the sample to be processed, the performance of the first deep learning model is better than the performance of the second deep learning model (good), or the performance of the first deep learning model is the same as the performance of the second deep learning model (same). For example, if the first evaluation index value is 8 and the second evaluation index value is 8, it can be determined that the sample evaluation result of the sample text to be processed is the first sample evaluation result.

[0168] In some embodiments, according to multiple evaluation index values, determining the sample evaluation result of the sample text to be processed includes: in response to determining that the first evaluation index value is less than the second evaluation index value, determining that the sample evaluation result of the sample text to be processed is the second sample evaluation result. The second sample evaluation result may indicate that when processing the sample text to be processed, the performance of the first deep learning model is worse (bad) than the performance of the second deep learning model. For example, if the first evaluation index value is 4 and the second evaluation index value is 8, it can be determined that the sample evaluation result of the sample text to be processed is the second sample evaluation result. Through the embodiments of the present disclosure, based on the comparison between the output results of different models, the model suitable for processing the sample text to be processed can be accurately determined, which helps to accurately determine the model applicable to processing new sample texts.

[0169] Next, operation S5021 may be performed to determine an acceptance rate.

[0170] In the embodiments of the present disclosure, the acceptance rate can indicate whether the performance of the first deep learning model in processing the sample text to be processed is acceptable. For example, taking a sample set to be processed including 500 sample texts to be processed as an example, the sample evaluation results of 400 sample texts to be processed in the sample set to be processed are the first sample evaluation results, and the sample evaluation results of 100 samples to be processed in the sample set to be processed are the second sample evaluation results. The acceptance rate of the sample set to be processed can be 80%. For another example, taking another sample set to be processed including 1,000 sample texts to be processed as an example, the sample evaluation results of 200 sample texts to be processed in the sample set to be processed are the first sample evaluation results, and the sample evaluation results of 800 samples to be processed in the sample set to be processed are the second sample evaluation results. The acceptance rate of the sample set to be processed can be 20%.

[0171] In an embodiment of the present disclosure, in response to determining that the acceptance rate is greater than or equal to the first preset acceptance rate threshold or the acceptance rate is less than or equal to the second preset acceptance rate threshold, operation S5022 is performed. For example, the first preset acceptance rate threshold may be 70%, and the second preset acceptance rate threshold may be 30%. If the acceptance rate of the above-mentioned one sample set to be processed is 80% (greater than the first preset acceptance rate threshold), operation S5022 may be performed on multiple sample texts to be processed in the sample set to be processed. For another example, if the acceptance rate of the above-mentioned other sample set to be processed is 20% (less than the second preset acceptance rate threshold), operation S5022 may be performed on multiple sample texts to be processed in the sample set to be processed. For another example, for one or more sample sets to be processed whose acceptance rate is less than the first preset acceptance rate threshold and greater than the second preset acceptance rate threshold, the sample texts to be processed in the one or more sample sets to be processed may be deleted.

[0172] In operation S5022, at least one feature tag is determined for each of the plurality of sample texts to be processed.

[0173] In the disclosed embodiment, at least one feature label of the sample text to be processed can be determined by using a large model according to the annotation prompt data. For example, at least one feature label of the sample text to be processed can be determined by using the above-mentioned annotation large model TAG30.

[0174] In the embodiment of the present disclosure, the sample text to be processed is used as the input sample text according to the preset template to be annotated.

[0175] In the disclosed embodiment, a preset text to be annotated is input into the large model so that the large model outputs at least one label data of the sample text to be processed according to the preset labeling result template. The label data includes a plurality of labeling result fields consistent with the preset labeling result template, the plurality of labeling result fields include a feature label field and a label explanation field, the value of the feature label field of the label data is the feature label, and the label explanation field of the label data is used to indicate the meaning of the feature label.

[0176] like Figure 5 As shown, based on multiple sample texts to be processed, multiple feature tags can be determined. The multiple feature tags may include "text analysis", "translation", "education", "policy interpretation", "keyword extraction", etc. It can be understood that Figure 5 The multiple feature tags shown are only examples. For one or more sets of samples to be processed whose acceptance rate is greater than or equal to a first preset acceptance rate threshold and one or more sets of samples to be processed whose acceptance rate is less than or equal to a second preset acceptance rate threshold, more than 2,000 different feature tags can be determined.

[0177] It can be understood that the above description of determining the first target feature label of the first target sample text is also applicable to determining the feature label of the sample text to be processed, and the present disclosure will not repeat it here. In addition, the above description of the annotation prompt data, the preset template to be annotated, and the preset annotation result template are also applicable to this embodiment, and the present disclosure will not repeat it here.

[0178] Next, operation S503 may be performed.

[0179] In operation S503, a feature evaluation value of a feature tag is determined according to respective sample evaluation results of a plurality of sample texts to be processed having the same feature tag.

[0180] In the embodiment of the present disclosure, feature evaluation data of each of the plurality of sample sets to be processed may be determined. The feature evaluation data of the sample set to be processed may include a set evaluation result of at least one feature label. The set evaluation result of the feature label may be determined based on the sample evaluation result of the sample text to be processed having the feature label in the sample set to be processed.

[0181] In the embodiment of the present disclosure, the set evaluation result may be a first set evaluation result or a second set evaluation result.

[0182] In an embodiment of the present disclosure, the set evaluation result of the feature label can be determined according to the sample evaluation results of the sample text to be processed with the feature label in the sample set to be processed by the following operation: in response to determining that the first sample evaluation ratio is greater than or equal to the first preset ratio threshold, the set evaluation result of the feature label is determined as the first set evaluation result. In the sample set to be processed, the first sample evaluation ratio is the ratio of the first sample text to be processed in the multiple sample texts to be processed with the feature label. The first sample text to be processed has a feature label. The sample evaluation result of the first sample text to be processed is the first sample evaluation result. For example, if there are multiple sample texts to be processed with the feature label "text analysis" in a sample set to be processed. The multiple samples to be processed with the feature label "text analysis" include at least one first sample text to be processed and at least one second sample text to be processed. The sample evaluation result of the second sample to be processed is the second sample evaluation result. In the set of samples to be processed, if the number of first sample texts to be processed with the feature label is 75, the number of second sample texts to be processed with the feature label is 25, and the first preset proportion threshold is 70%, it can be determined that the set evaluation result of the feature label "text analysis" is the first set evaluation result.

[0183] In another embodiment of the present disclosure, the set evaluation result of the feature label can be determined according to the sample evaluation results of the sample texts to be processed with the feature label in the sample set to be processed by the following operation: in response to determining that the first sample evaluation ratio is less than the second preset ratio threshold, the set evaluation result of the feature label is determined as the second set evaluation result. For example, there are also multiple sample texts to be processed with the feature label "text analysis" in another sample set to be processed. In the sample set to be processed, if the number of the first sample texts to be processed with the feature label "text analysis" is 25, the number of the second sample texts to be processed with the feature label "text analysis" is 75, and the second preset ratio threshold is 30%, the set evaluation result of the feature label "text analysis" can be determined as the second set evaluation result.

[0184] In an embodiment of the present disclosure, determining a feature evaluation value of a feature label according to respective sample evaluation results of a plurality of samples to be processed having the same feature label includes: determining the feature evaluation value of the feature label according to a first difference value and the number of target sample sets to be processed. The first difference value is determined according to a first number and a second number. The first number is the number of the first target sample set to be processed. The set evaluation result of the feature labels in the first target sample set to be processed is the first set evaluation result. The second number is the number of the second target sample set to be processed. The set evaluation result of the feature labels in the second target sample set to be processed is the second set evaluation result. For example, the above-mentioned two sample sets to be processed having the feature label "text analysis" can be used as two target sample sets to be processed. For another example, the feature evaluation value of the feature label can be determined by the following formula :

[0185] (Formula 2)

[0186] It can be the number of the first target sample set to be processed, It may be the number of the second target sample sets to be processed, and N may be the number of the target sample sets to be processed.

[0187] It can be understood that the feature evaluation value of the feature tag is determined according to the set evaluation result of the feature tag above, but the present disclosure is not limited thereto, and the feature evaluation value of the feature tag can also be determined according to the sample evaluation result, which will be explained below.

[0188] In the disclosed embodiment, in other implementations of the above operation S403, determining the feature evaluation value of the feature label according to the sample evaluation results of each of the multiple samples to be processed with the same feature label includes: determining the feature evaluation value of the feature label according to the second difference value and the number of the multiple sample texts to be processed with the same feature label. The second difference value is determined according to the third number and the fourth number. The third number is the number of the first sample evaluation results in the multiple sample texts to be processed with the same feature label. The fourth number is the number of the second sample evaluation results in the multiple sample texts to be processed with the same feature label. For example, for multiple sample texts to be processed with the feature label "keyword extraction", the third number of the sample texts to be processed whose sample evaluation results are the first sample evaluation results can be 300, and the fourth number of the sample texts to be processed whose sample evaluation results are the second sample evaluation results can be 100. The second difference value can be 200. The feature evaluation value of the feature label can be 0.5 (200 / 400).

[0189] It can be understood that some methods for determining the feature evaluation values ​​of feature tags are described above, and the sample evaluation results of the feature tags disclosed in the present invention will be further described below.

[0190] Figure 6 is a schematic diagram of multiple feature tags according to an embodiment of the present disclosure.

[0191] like Figure 6 As shown, the multiple feature tags may include "formatting", "resume analysis" and "text generation". For the multiple sample texts to be processed with the feature tag "formatting", the number of the first sample evaluation results is less than the number of the second sample evaluation results. For the multiple sample texts to be processed with the feature tag "resume analysis", the number of the first sample evaluation results is greater than the number of the second sample evaluation results. For the multiple sample texts to be processed with the feature tag "text review", the number of the first sample evaluation results is greater than the number of the second sample evaluation results.

[0192] As described above, after determining the feature evaluation value of the feature tag, the feature tag can be used as a preset feature tag, and the feature evaluation value of the feature tag can be used as a preset evaluation value. The sample text to be processed can also be used as a preset sample text, and the preset sample text will be described below. It can be understood that the processing method of the preset sample text is the same or similar to the processing method of the sample text to be processed, and the present disclosure will not repeat it here.

[0193] In some embodiments, the preset feature label is determined by using the macro model according to the labeling prompt data through the following operations: according to the preset template to be labeled, the preset sample text is used as the input sample text. The input sample text is input into the macro model so that the macro model outputs at least one preset label data of the preset sample text according to the preset labeling result template, the preset label data includes a plurality of labeling result fields consistent with the preset labeling result template, the plurality of labeling result fields include a feature label field and a label explanation field, the value of the feature label field of the preset label data is the preset feature label, and the label explanation field of the preset label data is used to indicate the meaning of the preset feature label.

[0194] In some embodiments, a preset feature label corresponds to a feature evaluation value, which is determined based on a sample evaluation result of each of a plurality of preset sample texts having the preset feature label, and a sample evaluation result of each of the plurality of preset sample texts is determined based on a plurality of model output results of each of the plurality of preset sample texts, and the plurality of model output results of the preset sample texts are respectively output by a plurality of deep learning models with different parameter amounts.

[0195] In some embodiments, the sample evaluation result of the preset sample text is determined based on multiple model output results of the preset sample text through the following operations: according to the evaluation prompt data, using the large model to respectively determine multiple evaluation index values ​​of each of the multiple model output results of the preset sample text; based on the multiple evaluation index values, determine the sample evaluation result of the preset sample text.

[0196] In some embodiments, a plurality of deep learning models with different parameter amounts include a first deep learning model and a second deep learning model, the parameter amount of the first deep learning model is less than the parameter amount of the second deep learning model, the plurality of model output results include a first model output result output by the first deep learning model and a second model output result output by the second deep learning model, the plurality of evaluation index values ​​include a first evaluation index value of the first model output result and a second evaluation index value of the second model output result, and according to the plurality of evaluation index values, determining a sample evaluation result of a preset sample text includes: in response to determining that the first evaluation index value is greater than or equal to the second evaluation index value, determining that the sample evaluation result of the preset sample text is a first sample evaluation result. In response to determining that the first evaluation index value is less than the second evaluation index value, determining that the sample evaluation result of the preset sample text is a second sample evaluation result.

[0197] In some embodiments, the feature evaluation value is determined according to the sample evaluation results of each of the plurality of preset sample texts with preset feature labels by the following operation: determining the feature evaluation value of the preset feature label according to the first difference value and the number of the target preset sample set, wherein the target preset sample set is a preset sample set including preset sample texts with preset feature labels, the first difference value is determined according to the first number and the second number, the first number is the number of the first target preset sample set, the set evaluation result of the preset feature labels in the first target preset sample set is the first set evaluation result, the second number is the number of the second target preset sample set, the set evaluation result of the feature labels in the second target preset sample set is the second set evaluation result. The set evaluation result is determined according to the sample evaluation results of the preset sample texts with preset feature labels in the preset sample set, and the preset sample set is obtained by clustering the plurality of preset sample texts.

[0198] In some embodiments, the set evaluation result of the feature tag is determined according to the sample evaluation results of the sample text to be processed with the feature tag in the sample set to be processed by the following operation: in response to determining that the first sample evaluation ratio is greater than or equal to the first preset ratio threshold, the set evaluation result of the preset feature tag is determined as the first set evaluation result, wherein the first sample evaluation ratio is the ratio of the first preset sample text in the multiple preset sample texts with the preset feature tag in the preset sample set, the first preset sample text has the preset feature tag, and the sample evaluation result of the first preset sample text is the first sample evaluation result. In response to determining that the first sample evaluation ratio is less than the second preset ratio threshold, the set evaluation result of the preset feature tag is determined as the second set evaluation result.

[0199] In some embodiments, the feature evaluation value is determined according to the sample evaluation results of each of the plurality of preset sample texts with the preset feature label by the following operation: according to the second difference value and the number of preset sample texts with the preset feature label, the feature evaluation value of the preset feature label is determined. The second difference value is determined according to the third number and the second number, the first number being the number of the first evaluation results in the plurality of preset sample texts with the same feature label, and the second number being the number of the second evaluation results in the plurality of preset sample texts with the same feature label.

[0200] In some embodiments, the plurality of evaluation prompt texts include evaluation configuration prompt text, accuracy evaluation prompt text, instruction compliance prompt text, directness prompt text, fluency evaluation prompt text, compliance evaluation prompt text, consistency evaluation prompt text, and evaluation result prompt text. The evaluation configuration prompt text is used to indicate the evaluation index value of the evaluation large model to determine the model output result. The accuracy evaluation prompt text is used to indicate the accuracy of the evaluation large model to determine the model output result. The instruction compliance evaluation prompt text is used to indicate whether the evaluation large model determines whether the model output result is generated for a preset sample text. The directness evaluation prompt text is used to indicate whether the evaluation large model determines whether the model output result includes information unrelated to the preset sample text. The fluency evaluation prompt text is used to indicate whether the evaluation large model determines whether the model output result meets the preset grammatical conditions. The compliance evaluation prompt text is used to indicate whether the evaluation large model determines whether the model output result meets the preset rules. The consistency evaluation prompt text is used to indicate whether the model output result is consistent with the reference information in the preset sample text. The evaluation result prompt text is used to indicate the numerical range of the evaluation index value.

[0201] In some embodiments, according to the evaluation prompt data, using the large model to respectively determine the multiple evaluation index values ​​of the multiple model output results of the preset sample text includes: according to the preset template to be evaluated, the preset sample text and the multiple model output results are spliced ​​into the text to be evaluated. The text to be evaluated is input into the large model so that the large model outputs the evaluation data of the preset sample text according to the preset evaluation result template. The evaluation data includes multiple evaluation result fields consistent with the preset evaluation result template, and the values ​​of the evaluation result fields are evaluation index values.

[0202] It can be understood that the method of the present disclosure is described above, and the device of the present disclosure will be described below.

[0203] Figure 7 is a block diagram of a model determination device according to an embodiment of the present disclosure.

[0204] like Figure 7 As shown, the apparatus 700 may include a first determining module 710 and a second determining module 720 .

[0205] The first determination module 710 is used to determine the model evaluation value of the target scene according to at least one feature evaluation value corresponding to at least one first target feature label of the first target sample text, where the first target sample text is the sample text of the target scene.

[0206] The second determination module 720 is used to determine a target model from a plurality of deep learning models with different parameter amounts according to the model evaluation value. The target model is used to process the question text of the target scenario.

[0207] In some embodiments, the first determination module includes: a first determination submodule, configured to determine at least one first target feature tag of the first target sample text using the large model according to the annotation prompt data; and a second determination submodule, configured to determine the model evaluation value according to at least one feature evaluation value of the at least one first target feature tag.

[0208] In some embodiments, the first determination submodule includes: a first obtaining unit, used to take the first target sample text as the input sample text according to a preset template to be annotated. A first input unit, used to input the input sample text target text to be annotated into the large model, so that the large model outputs at least one target label data of the first target sample text according to the preset annotation result template. The target label data includes a plurality of annotation result fields consistent with the preset annotation result template, the plurality of annotation result fields include a feature label field and a label explanation field, the value of the feature label field of the target label data is the first target feature label, and the label explanation field of the target label data is used to indicate the meaning of the first target feature label.

[0209] In some embodiments, the plurality of deep learning models with different parameter amounts include a first deep learning model and a second deep learning model, and the parameter amount of the first deep learning model is less than the parameter amount of the second deep learning model. The second determination module includes: a third determination submodule, for determining the first deep learning model as a target model in response to determining that the model evaluation value is greater than or equal to a preset evaluation threshold.

[0210] In some embodiments, the device further comprises: a processing module, configured to process the question text of the target scenario using the fine-tuned target model to obtain a reply text corresponding to the question text. The fine-tuned target model is obtained by fine-tuning the target model using at least one training sample data, the at least one training sample data comprising at least one of a first target sample text and a second target sample text, the second target sample text being determined from a plurality of preset sample texts according to at least one first target feature label.

[0211] In some embodiments, the first target feature label is determined from a plurality of preset feature labels, the preset feature label is determined using a large model based on the annotation prompt data, and the preset feature label corresponds to a feature evaluation value.

[0212] In some embodiments, the preset feature label is determined by using the macro model according to the labeling prompt data through the following operations: according to the preset template to be labeled, the preset sample text is used as the input sample text. The input sample text is input into the macro model so that the macro model outputs at least one preset label data of the preset sample text according to the preset labeling result template, the preset label data includes a plurality of labeling result fields consistent with the preset labeling result template, the plurality of labeling result fields include a feature label field and a label explanation field, the value of the feature label field of the preset label data is the preset feature label, and the label explanation field of the preset label data is used to indicate the meaning of the preset feature label.

[0213] In some embodiments, the annotation prompt data includes at least one of a plurality of annotation prompt texts, and the annotation prompt text is used to instruct the large model to perform an annotation operation corresponding to the annotation prompt text.

[0214] In some embodiments, the multiple annotation prompt texts include annotation configuration prompt text, annotation type prompt text, first annotation restriction prompt text, second annotation restriction prompt text, and annotation order prompt text. The annotation configuration prompt text is used to instruct the large model to determine the feature label of the input sample text based on the keywords of the input sample text. The annotation type prompt text is used to provide definitions of different types of keywords. The first annotation restriction prompt text is used to indicate keywords that the large model does not need to extract from the input sample text. The second annotation restriction prompt text is used to instruct the large model to determine keywords from multiple initial words of the input sample text. The annotation order prompt text is used to instruct the large model to output keywords according to the text position corresponding to the keywords, and the text position is the position corresponding to the keyword in the input sample text.

[0215] In some embodiments, the annotation prompt text includes at least one of a reference information sub-text, an identification sub-text, and a core sub-text. The reference information sub-text is used to provide reference information for the large model. The core sub-text includes core words and core tags. The core words are used to indicate operations that the large model needs to perform or does not need to perform.

[0216] In some embodiments, a preset feature label corresponds to a feature evaluation value, and the feature evaluation value is determined based on sample evaluation results of each of a plurality of preset sample texts having the preset feature label. The sample evaluation results of each of the plurality of preset sample texts are determined based on a plurality of model output results of each of the plurality of preset sample texts, and the plurality of model output results of the preset sample texts are respectively output by a plurality of deep learning models with different parameter amounts.

[0217] In some embodiments, the sample evaluation result of the preset sample text is determined by the following submodules performing related operations based on multiple model output results of the preset sample text: a fourth determination submodule, used to determine multiple evaluation index values ​​of each of the multiple model output results of the preset sample text using the large model according to the evaluation prompt data. A fifth determination submodule, used to determine the sample evaluation result of the preset sample text according to the multiple evaluation index values.

[0218] In some embodiments, a plurality of deep learning models with different parameter amounts include a first deep learning model and a second deep learning model, and the parameter amount of the first deep learning model is less than the parameter amount of the second deep learning model. The plurality of model output results include a first model output result output by the first deep learning model and a second model output result output by the second deep learning model. The plurality of evaluation index values ​​include a first evaluation index value of the first model output result and a second evaluation index value of the second model output result. The fifth determination submodule includes: a first determination unit, for determining that a sample evaluation result of a preset sample text is a first sample evaluation result in response to determining that the first evaluation index value is greater than or equal to the second evaluation index value. A second determination unit, for determining that a sample evaluation result of a preset sample text is a second sample evaluation result in response to determining that the first evaluation index value is less than the second evaluation index value.

[0219] In some embodiments, the feature evaluation value is determined by performing relevant operations according to the sample evaluation results of each of multiple preset sample texts with preset feature labels by the following submodules: a sixth determination submodule, which is used to determine the feature evaluation value of the preset feature label according to the first difference value and the number of the target preset sample set. The target preset sample set is a preset sample set including preset sample texts with preset feature labels, and the first difference value is determined according to the first number and the second number, the first number is the number of the first target preset sample set, and the set evaluation result of the preset feature labels in the first target preset sample set is the first set evaluation result, the second number is the number of the second target preset sample set, and the set evaluation result of the feature labels in the second target preset sample set is the second set evaluation result. The set evaluation result is determined according to the sample evaluation results of the preset sample texts to be processed with the preset feature labels in the preset sample set to be processed, and the preset sample set is obtained by clustering multiple preset sample set texts.

[0220] In some embodiments, the set evaluation result of the feature label is determined by performing relevant operations according to the sample evaluation results of the preset sample texts to be processed with the feature label in the preset sample set to be processed by the following submodules: a sixth determination submodule, which is used to determine the set evaluation result of the preset feature label as the first set evaluation result in response to determining that the first sample evaluation ratio is greater than or equal to the first preset ratio threshold. The first sample evaluation ratio is the ratio of the first preset sample text to be processed in the multiple sample texts to be processed with the preset feature label in the preset sample set to be processed, the first preset sample text to be processed has the preset feature label, and the sample evaluation result of the first preset sample text is the sample text to be processed of the first sample evaluation result. A seventh determination submodule, which is used to determine the set evaluation result of the preset feature label as the second set evaluation result in response to determining that the first sample evaluation ratio is less than the second preset ratio threshold.

[0221] In some embodiments, the feature evaluation value is determined by performing relevant operations according to the sample evaluation results of each of the plurality of preset sample texts with the preset feature label by the following submodules: an eighth determination submodule, for determining the feature evaluation value of the feature label according to the second difference value and the number of the plurality of preset sample texts with the same feature label. The second difference value is determined according to the third number and the fourth number, the third number being the number of the first sample evaluation results in the plurality of preset sample texts with the same feature label, and the fourth number being the number of the second sample evaluation results in the plurality of preset sample texts with the same feature label.

[0222] In some embodiments, the evaluation prompt data text includes at least one of a plurality of evaluation prompt texts, and the evaluation prompt text is used to instruct the evaluation large model to evaluate the model output result according to the evaluation method corresponding to the evaluation prompt text.

[0223] In some embodiments, the plurality of evaluation prompt texts include evaluation configuration prompt text, accuracy evaluation prompt text, instruction compliance prompt text, directness prompt text, fluency evaluation prompt text, compliance evaluation prompt text, consistency evaluation prompt text, and evaluation result prompt text. The evaluation configuration prompt text is used to indicate the evaluation index value of the evaluation large model to determine the model output result. The accuracy evaluation prompt text is used to indicate the accuracy of the evaluation large model to determine the model output result. The instruction compliance evaluation prompt text is used to indicate whether the evaluation large model determines whether the model output result is generated for a preset sample text. The directness evaluation prompt text is used to indicate whether the evaluation large model determines whether the model output result includes information unrelated to the preset sample text. The fluency evaluation prompt text is used to indicate whether the evaluation large model determines whether the model output result meets the preset grammatical conditions. The compliance evaluation prompt text is used to indicate whether the evaluation large model determines whether the model output result meets the preset rules. The consistency evaluation prompt text is used to indicate whether the model output result is consistent with the reference information in the preset sample text. The evaluation result prompt text is used to indicate the numerical range of the evaluation index value.

[0224] In some embodiments, the fourth determination submodule includes: a splicing unit, which is used to splice the preset sample text and the output results of multiple models into the text to be evaluated according to the preset template to be evaluated. A second input unit is used to input the text to be evaluated into the large model so that the large model outputs the evaluation data of the preset sample text according to the preset evaluation result template. The evaluation data includes multiple evaluation result fields consistent with the preset evaluation result template, and the value of the evaluation result field is the evaluation index value.

[0225] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0226] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0227] like Figure 8As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0228] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0229] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the model determination method. For example, in some embodiments, the model determination method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the model determination method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the model determination method in any other appropriate manner (eg, by means of firmware).

[0230] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0231] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0232] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories, read-only memories, erasable programmable read-only memories (EPROM) or flash memories, optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0233] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) display or a liquid crystal display (LCD)) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0234] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a Local Area Network (LAN), a Wide Area Network (WAN), and the Internet.

[0235] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0236] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0237] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A model determination method, comprising: Determining a model evaluation value of the target scene according to at least one feature evaluation value corresponding to at least one first target feature label of a first target sample text, wherein the first target sample text is a sample text of the target scene; According to the model evaluation value, a target model is determined from a plurality of deep learning models with different parameter amounts, wherein the target model is used to process the question text of the target scenario.

2. The method according to claim 1, wherein: Determining the model evaluation value of the target scene by using at least one feature evaluation value corresponding to at least one first target feature label of the first target sample text includes: Determine at least one first target feature label of the first target sample text using the large model according to the annotation prompt data; The model evaluation value is determined based on at least one feature evaluation value of at least one of the first target feature tags.

3. The method according to claim 2, wherein: Determining at least one of the first target feature labels of the first target sample text using the large model according to the annotation prompt data includes: According to the preset template to be annotated, the first target sample text is used as the input sample text; The input sample text is input into the big model so that the big model outputs at least one target label data of the first target sample text according to a preset labeling result template, wherein the target label data includes a plurality of labeling result fields consistent with the preset labeling result template, the plurality of labeling result fields include a feature label field and a label explanation field, the value of the feature label field of the target label data is the first target feature label, and the label explanation field of the target label data is used to indicate the meaning of the first target feature label.

4. The method according to claim 1, wherein: The multiple deep learning models with different parameter amounts include a first deep learning model and a second deep learning model, the parameter amount of the first deep learning model is smaller than the parameter amount of the second deep learning model, Determining a target model from a plurality of deep learning models with different parameter amounts according to the model evaluation value comprises: In response to determining that the model evaluation value is greater than or equal to a preset evaluation threshold, the first deep learning model is determined as the target model.

5. The method according to claim 1, further comprising: Processing the question text of the target scenario using the fine-tuned target model to obtain a response text corresponding to the question text, Wherein, the fine-tuned target model is obtained by fine-tuning the target model using at least one training sample data, and the at least one training sample data includes at least one of the first target sample text and the second target sample text, and the second target sample text is determined from multiple preset sample texts based on at least one of the first target feature labels.

6. The method according to claim 1, wherein: The first target feature label is determined from a plurality of preset feature labels, the preset feature label is determined using a large model according to the annotation prompt data, and the preset feature label corresponds to one of the feature evaluation values.

7. According to the method of claim 6, the preset feature label is determined by using the large model according to the annotation prompt data by the following operations: According to the preset template to be annotated, the preset sample text is used as the input sample text; The input sample text is input into the big model so that the big model outputs at least one preset label data of the preset sample text according to a preset labeling result template, the preset label data includes a plurality of labeling result fields consistent with the preset labeling result template, the plurality of labeling result fields include a feature label field and a label explanation field, the value of the feature label field of the preset label data is the preset feature label, and the label explanation field of the preset label data is used to indicate the meaning of the preset feature label.

8. The method according to claim 2 or 6, wherein: The annotation prompt data includes at least one of a plurality of annotation prompt texts, and the annotation prompt text is used to instruct the large model to perform an annotation operation corresponding to the annotation prompt text.

9. The method according to claim 8, wherein: The plurality of annotation prompt texts include annotation configuration prompt text, annotation type prompt text, first annotation restriction prompt text, second annotation restriction prompt text and annotation order prompt text, The annotation configuration prompt text is used to instruct the large model to determine the feature label of the input sample text according to the keywords of the input sample text. The annotation type prompt text is used to provide definitions of different types of keywords. The first annotation restriction prompt text is used to indicate that the large model does not need to extract keywords from the input sample text. The second annotation restriction prompt text is used to instruct the large model to determine keywords from a plurality of initial words in the input sample text. The annotation order prompt text is used to instruct the large model to output the keyword according to the text position corresponding to the keyword, and the text position is the position corresponding to the keyword in the input sample text.

10. The method according to claim 8, wherein: The annotation prompt text includes at least one of a reference information subtext, an identification subtext and a core subtext. The reference information subtext is used to provide reference information for the large model. The core subtext includes core words and core tags. The core words are used to indicate operations that the large model needs to perform or does not need to perform.

11. The method according to claim 6, wherein: The preset feature label corresponds to one feature evaluation value, and the feature evaluation value is determined according to sample evaluation results of each of a plurality of preset sample texts having the preset feature label. The sample evaluation results of each of the plurality of preset sample texts are determined based on the plurality of model output results of each of the plurality of preset sample texts, and the plurality of model output results of the preset sample texts are respectively output by the deep learning models with a plurality of different parameter amounts.

12. The method according to claim 11, wherein: The sample evaluation result of the preset sample text is determined according to the output results of multiple models of the preset sample text by the following operations: According to the evaluation prompt data, using the large model, respectively determine a plurality of evaluation index values ​​of the plurality of model output results of the preset sample text; According to the multiple evaluation index values, a sample evaluation result of the preset sample text is determined.

13. The method according to claim 12, wherein: The multiple deep learning models with different parameter amounts include a first deep learning model and a second deep learning model, wherein the parameter amount of the first deep learning model is smaller than the parameter amount of the second deep learning model, The multiple model output results include a first model output result output by the first deep learning model and a second model output result output by the second deep learning model, The plurality of evaluation index values ​​include a first evaluation index value of an output result of the first model and a second evaluation index value of an output result of the second model, Determining the sample evaluation result of the preset sample text according to the multiple evaluation index values ​​includes: In response to determining that the first evaluation index value is greater than or equal to the second evaluation index value, determining the sample evaluation result of the preset sample text as a first sample evaluation result; In response to determining that the first evaluation index value is less than the second evaluation index value, determining the sample evaluation result of the preset sample text as the second sample evaluation result.

14. The method according to claim 13, wherein: The feature evaluation value is determined according to respective sample evaluation results of a plurality of preset sample texts having the preset feature label by the following operations: According to the first difference value and the number of the target preset sample set, the feature evaluation value of the preset feature label is determined, wherein the target preset sample set is the preset sample set including the preset sample text with the preset feature label, the first difference value is determined according to the first number and the second number, the first number is the number of the first target preset sample set, the set evaluation result of the preset feature label in the first target preset sample set is the first set evaluation result, the second number is the number of the second target preset sample set, and the set evaluation result of the feature label in the second target preset sample set is the second set evaluation result, The set evaluation result is determined according to the sample evaluation results of the preset sample texts with the preset feature labels in the preset sample set, and the preset sample set is obtained by clustering a plurality of the preset sample texts.

15. The method according to claim 14, wherein: The set evaluation result of the feature label is determined according to the sample evaluation results of the preset sample texts with the feature label in the preset sample set by the following operations: In response to determining that the first sample evaluation ratio is greater than or equal to a first preset ratio threshold, determining the set evaluation result of the preset feature label as the first set evaluation result, wherein the first sample evaluation ratio is the ratio of the first preset sample text in the plurality of to-be-processed sample texts having the preset feature label in the preset sample set, the first preset sample text has the preset feature label, and the sample evaluation result of the first preset sample text is the first sample evaluation result; In response to determining that the first sample evaluation ratio is less than a second preset ratio threshold, the set evaluation result of the preset feature label is determined as the second set evaluation result.

16. The method according to claim 13, wherein: The feature evaluation value is determined according to respective sample evaluation results of a plurality of preset sample texts having the preset feature label by the following operations: A feature evaluation value of the feature label is determined based on a second difference value and the number of multiple preset sample texts with the same feature label, wherein the second difference value is determined based on a third number and a fourth number, the third number being the number of first sample evaluation results in the multiple preset sample texts with the same feature label, and the fourth number being the number of second sample evaluation results in the multiple preset sample texts with the same feature label.

17. The method according to claim 12, wherein: The evaluation prompt data text includes at least one of a plurality of evaluation prompt texts, and the evaluation prompt text is used to instruct the evaluation large model to evaluate the model output result according to the evaluation method corresponding to the evaluation prompt text.

18. The method according to claim 17, wherein: The plurality of evaluation prompt texts include evaluation configuration prompt text, accuracy evaluation prompt text, instruction compliance prompt text, directness prompt text, fluency evaluation prompt text, compliance evaluation prompt text, consistency evaluation prompt text, and evaluation result prompt text, The evaluation configuration prompt text is used to indicate the evaluation index value of the model output result determined by the evaluation large model. The accuracy assessment prompt text is used to instruct the evaluation model to determine the accuracy of the model output result. The instruction follows the evaluation prompt text for instructing the evaluation model to determine whether the model output result is generated for the preset sample text. The directness evaluation prompt text is used to instruct the evaluation model to determine whether the model output result includes information unrelated to the preset sample text. The fluency evaluation prompt text is used to instruct the evaluation model to determine whether the model output result meets the preset grammatical conditions. The compliance assessment prompt text is used to instruct the assessment model to determine whether the model output result complies with the preset rules. The consistency assessment prompt text is used to indicate whether the model output result is consistent with the reference information in the preset sample text; The evaluation result prompt text is used to indicate the numerical range of the evaluation index value.

19. The method according to claim 15, wherein: The method of using the large model to respectively determine the multiple evaluation index values ​​of the multiple model output results of the preset sample text according to the evaluation prompt data includes: According to the preset template to be evaluated, the preset sample text and the output results of the multiple models are spliced ​​into the text to be evaluated; The text to be evaluated is input into the big model so that the big model outputs evaluation data of the preset sample text according to a preset evaluation result template, wherein the evaluation data includes multiple evaluation result fields consistent with the preset evaluation result template, and the value of the evaluation result field is the evaluation index value.

20. A model determination device, comprising: A first determination module, configured to determine a model evaluation value of the target scene according to at least one feature evaluation value corresponding to at least one first target feature tag of a first target sample text, wherein the first target sample text is a sample text of the target scene; The second determination module is used to determine a target model from a plurality of deep learning models with different parameter amounts according to the model evaluation value, wherein the target model is used to process the question text of the target scenario.

21. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 19.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 19.

23. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 19.

Images