A method and system for selecting and building a large language model based on task orientation

Abstract

The present invention belongs to the field of machine learning and enterprise risk control, and specifically relates to a method and system for selecting and constructing a task-oriented large language model, including: presetting multiple large language model task fields, and predefining corresponding inputs, evaluation indicators and outputs for different large language model task fields; receiving a user query request, and using a Transformer-based classifier model to identify the task type to which the request belongs; dynamically selecting a large language model adapted to the field for modeling, and combining corpus data of the corresponding task field to construct a large language model evaluation standard; performing automated testing and quantitative scoring on multiple candidate large language models, and performing model screening and weight allocation based on the scoring results to construct an optimal large language model combination; inputting the user query request into the optimal large language model combination, weighting the output probability of each model, and using autoregressive generation technology to sample word by word to generate a final text output.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning and enterprise risk control, and specifically relates to a method and system for selecting and building a task-oriented large language model. Background Art

[0002] With the rapid development of artificial intelligence and machine learning technologies, large language models have made breakthrough progress in the field of natural language processing. In the field of enterprise risk control, large language models can understand and generate human language through deep learning and training with large amounts of text data, thereby more efficiently identifying risks, monitoring public opinion, and assessing credit risks. In actual applications, the diversity of task requirements and application scenarios makes the selection of large language models the key to improving enterprise risk control capabilities.

[0003] Traditional large language model selection methods have limitations, the evaluation mechanism lacks systematization and standardization, and the comprehensiveness and consistency of the test cannot be guaranteed, resulting in high subjectivity and randomness in the model selection process. Faced with large-scale data and multi-task scenarios, the evaluation efficiency of existing methods is low and cannot quickly respond to actual application needs, which is particularly evident in the field of enterprise risk control. Existing technical means have many shortcomings in model selection and evaluation, which restricts the performance of large language models in actual enterprise risk control applications and cannot meet the complex needs of actual applications. Summary of the invention

[0004] The present invention provides a method and system for selecting and constructing a large language model based on task orientation. The method presets the task field of the large language model, identifies the task type and evaluates the model based on user query requests, selects the optimal large language model combination through quantitative scoring screening, and then uses autoregressive generation technology to sample word by word based on user query requests to generate high-quality and coherent text output. This method can improve the intelligence level of model selection, while ensuring the accuracy and fluency of the output text, and provide high-quality text generation services for enterprise risk control.

[0005] A method for selecting and building a task-oriented large language model includes the following steps:

[0006] Preset multiple large language model task fields, and predefine corresponding input, evaluation indicators and output formats for different large language model task fields;

[0007] Receive user query requests and use the Transformer-based classifier model to automatically identify the task type to which the request belongs;

[0008] Based on the identified task type, dynamically select a large language model that is suitable for the field for modeling, and build a large language model evaluation standard based on the corpus data of the corresponding task field;

[0009] Based on the constructed large language model evaluation criteria, multiple candidate large language models are automatically tested and quantitatively scored, and the candidate large language models are screened and weighted based on the scoring results to construct the optimal large language model combination;

[0010] The user query request is used as the input of the optimal large language model combination, the output probability of each model in the optimal large language model combination is weighted and fused, and the autoregressive generation technology is used to sample word by word to generate the final text output.

[0011] By presetting the task domain of the large language model, identifying the task type and evaluating the model based on user query requests, selecting the optimal large language model combination through quantitative scoring screening, and then using autoregressive generation technology to sample word by word based on user query requests to generate high-quality and coherent text output, the intelligence level of model selection can be improved, while ensuring the accuracy and fluency of the output text, providing high-quality text generation services for enterprise risk control.

[0012] Furthermore, the large language model task areas include text generation tasks, semantic understanding tasks, information extraction tasks, and translation tasks.

[0013] Furthermore, the process of presetting multiple large language model task domains and predefining corresponding input, evaluation indicators and output formats for different large language model task domains includes the following steps:

[0014] Based on the text generation task, a group of unfinished sentences or paragraphs are used as input data of the large language model, and text generation is performed in combination with the evaluation indicators of the text generation task, and a completed text paragraph is output; the evaluation indicators of the text task generation include fluency evaluation, coherence evaluation and accuracy evaluation;

[0015] Based on the semantic understanding task, two text segments to be compared are used as input data of the large language model, and semantic similarity is judged in combination with the evaluation index of the semantic understanding task, and a similarity score or a matching label is output; the evaluation index of the semantic understanding task includes semantic matching evaluation and accuracy evaluation;

[0016] Based on the information extraction task, a text document containing annotated information is used as input data of a large language model, structured information is extracted in combination with evaluation indicators of the information extraction task, and a list of extracted structured information is output; the evaluation indicators of the information extraction task include extraction accuracy evaluation and recall rate evaluation;

[0017] Based on the translation task, the original sentence is used as the input data of the large language model, and the translation quality is evaluated in combination with the evaluation indicators of the translation task, and the translation text in the target language is output; the evaluation indicators of the translation task include translation accuracy evaluation and naturalness evaluation;

[0018] The fluency evaluation and the naturalness evaluation are measured by perplexity; the coherence evaluation and the translation accuracy evaluation are measured by BLEU score; and the accuracy evaluation is measured by ROUGE-L indicator.

[0019] By predefining text generation tasks, the generated text can have contextual logic and natural language expression, and the output text can be consistent with the input text in terms of content and style; by predefining semantic understanding tasks, the semantic relationship between text fragments can be accurately judged; by predefining information extraction tasks, the specified information can be accurately and comprehensively extracted; by predefining translation tasks, the translated text can accurately express the meaning of the original text and have natural and fluent language expression; by predefining the input, evaluation indicators and output format of each task field, the comprehensiveness and consistency of the test can be ensured, thereby providing a standardized basis for subsequent model selection and construction, and systematically evaluating the performance of large language models on different tasks, so as to select the optimal large language model combination.

[0020] Furthermore, the process of receiving a user query request and automatically identifying the task type to which the request belongs using a Transformer-based classifier model specifically includes the following steps:

[0021] Receive user query requests and use them as input to the Transformer model. After being processed by multiple encoder layers of the Transformer model, the contextual semantic information of the user query request is output.

[0022] The obtained contextual semantic information is used as the input of the classifier layer, and the classifier layer processes and outputs the probability distribution of each task category;

[0023] Based on the probability distribution of each task category, the task category with the highest probability is selected as the most relevant task type.

[0024] By processing the encoder layer and the classifier layer in the Transformer classifier model, the most relevant task type is output, which can ensure that the subsequent model selection and construction are based on the correct task execution.

[0025] Furthermore, based on the identified task type, a large language model adapted to the field is dynamically selected for modeling, and the process of constructing a large language model evaluation standard in combination with the corpus data of the corresponding task field specifically includes the following steps:

[0026] Based on the corpus, extracting corpus text data associated with the identified task type;

[0027] The extracted corpus text data is used as the input of the large language model, and the evaluation score of each corpus text data is obtained. The comprehensive performance score under the corresponding task type is calculated to complete the construction of the large language model evaluation standard.

[0028] By automatically constructing large language model evaluation standards, a reliable basis can be provided for subsequent model selection.

[0029] Furthermore, based on the constructed large language model evaluation criteria, multiple candidate large language models are automatically tested and quantitatively scored, and the candidate large language models are screened and weighted based on the scoring results. The process of constructing the optimal large language model combination specifically includes the following steps:

[0030] Select multiple candidate large language models;

[0031] Based on the same test template and corpus text data, they are input into multiple candidate large language models for automated testing and quantitative scoring;

[0032] Based on the quantitative scoring results, the candidate large language models are sorted and screened;

[0033] Based on the selected large language models, the model weights are set in combination with the current task performance to build the optimal large language model combination.

[0034] By setting the same test template and corpus text data, we ensure that only the model itself is tested, and set the model weight based on task performance, so that we can provide a reference for building the optimal large language model combination.

[0035] Furthermore, the user query request is used as the input of the optimal large language model combination, the output probability of each model in the optimal large language model combination is weighted and fused, and the process of generating the final text output by sampling word by word using the autoregressive generation technology specifically includes the following steps:

[0036] The user query request is used as the input of each of the selected language models to generate the corresponding output probability distribution;

[0037] Based on the output probability distribution of each language model, the corresponding model weights are combined to perform weighted fusion processing to output the output probability of the optimal large language model combination;

[0038] The autoregressive generation technology is used to generate each word, and the output probability of the optimal large language model combination is sampled to generate the final text output.

[0039] By adopting the word-by-word generation and sampling method of autoregressive generation technology, it is possible to generate coherent and natural text output, ensuring that users receive the best quality text generation service.

[0040] A system for selecting and building a task-oriented large language model, comprising:

[0041] A task domain setting module, which is used to preset multiple large language model task domains and predefine corresponding input, evaluation indicators and output formats for different large language model task domains;

[0042] The task type identification module is used to receive user query requests and automatically identify the task type to which the request belongs using a Transformer-based classifier model;

[0043] A model evaluation standard construction module is used to dynamically select a large language model suitable for the field for modeling based on the identified task type, and to construct a large language model evaluation standard in combination with the corpus data of the corresponding task field;

[0044] The model screening and construction module is used to automatically test and quantitatively score multiple candidate large language models based on the constructed large language model evaluation criteria, and to screen and weight the candidate large language models based on the scoring results to build the optimal large language model combination;

[0045] The model output module is used to take the user query request as the input of the optimal large language model combination, perform weighted fusion processing on the output probability of each model in the optimal large language model combination, and use autoregressive generation technology to sample word by word to generate the final text output.

[0046] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0047] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0048] The beneficial effects of the present invention are:

[0049] The present invention presets the task field of the large language model, identifies the task type and evaluates the model based on the user query request, selects the optimal large language model combination through quantitative scoring screening, and then uses the autoregressive generation technology to sample word by word based on the user query request to generate high-quality and coherent text output. It can improve the intelligence level of model selection while ensuring the accuracy and fluency of the output text, providing high-quality text generation services for enterprise risk control. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of the present invention;

[0051] Figure 2 It is a structural schematic diagram of the system in the present invention;

[0052] Figure 3 It is a schematic diagram of the structure of the computer device in the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0055] In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples, and a person skilled in the art can understand the specific meanings of the above terms in this application under specific circumstances.

[0056] Example 1

[0057] Figure 1 The method for selecting and building a large language model based on task orientation is shown. By presetting the task domain of the large language model, identifying the task type and evaluating the model based on user query requests, the optimal large language model combination is selected through quantitative scoring screening, and then the autoregressive generation technology is used to sample each word based on the user query request to generate high-quality and coherent text output, which can improve the intelligent level of model selection, while ensuring the accuracy and fluency of the output text, and provide high-quality text generation services for enterprise risk control. The specific steps include the following:

[0058] S1: Preset multiple large language model task fields, and predefine corresponding input, evaluation indicators and output formats for different large language model task fields;

[0059] In this embodiment, the large language model task domain includes text generation tasks, semantic understanding tasks, information extraction tasks, and translation tasks.

[0060] Specifically, based on the text generation task, the process of predefining the corresponding input, evaluation indicators and output format includes the following steps:

[0061] S111: Based on diverse text fragments from different fields, a set of unfinished sentences or paragraphs are used as input data for a large language model;

[0062] In this embodiment, the diverse text segments in different fields include news reports, literary works, technical documents, and the like.

[0063] S112: Combined with the evaluation indicators of the text generation task, a large language model is used for text generation;

[0064] In this embodiment, the evaluation indicators generated by the text task include fluency assessment, coherence assessment, and accuracy assessment;

[0065] Among them, the fluency evaluation is measured by perplexity, and its expression is:

[0066] ;

[0067] In the formula, Represents the perplexity value, i.e., the fluency evaluation result; Indicates the number of data. ; Indicates the total number of data in the input data set; Indicates The weight of the data; Indicates the weight of the first data;

[0068] Among them, the coherence evaluation is measured by BLEU score, which is expressed as:

[0069] ;

[0070] In the formula, express Score value, i.e., coherence assessment result; Represents the length penalty term, which is used to penalize shorter candidate texts; Indicates the n-gram exact match rate;

[0071] Among them, the accuracy evaluation is measured by the ROUGE-L indicator, and its expression is:

[0072] ;

[0073] In the formula, It represents the ROUGE-L index value, i.e., the accuracy evaluation result; Represents text and text The longest common subsequence of ; Represents text and text Maximum length;

[0074] S113: Outputting a completed text paragraph, wherein the completed text paragraph is consistent with the input data in terms of content and style.

[0075] Specifically, based on the semantic understanding task, the process of predefining the corresponding input, evaluation indicators and output format includes the following steps:

[0076] S121: using two text segments to be compared as input data of the large language model;

[0077] In this embodiment, the two text segments to be compared have different degrees of semantic relevance.

[0078] S122: Combined with the evaluation indicators of the semantic understanding task, a large language model is used to determine the semantic similarity;

[0079] In this embodiment, the evaluation indicators of the semantic understanding task include semantic matching evaluation and accuracy evaluation;

[0080] Among them, the expression of semantic matching evaluation is:

[0081] ;

[0082] In the formula, Represents a text fragment and text snippets The semantic matching value of , i.e., the semantic matching evaluation result; Represents a semantic unit, ; Indicates the total number of semantic units; Indicates The matching score of the semantic units;

[0083] Among them, the expression of accuracy evaluation is:

[0084] ;

[0085] In the formula, Indicates the accuracy value, i.e., the accuracy evaluation result; represents correctly identified matching pairs; represents misidentified matching pairs; represents an unidentified matching pair;

[0086] S123: Output similarity score or matching label;

[0087] In this embodiment, the matching degree labels include similarity and dissimilarity.

[0088] Specifically, based on the information extraction task, the process of predefining the corresponding input, evaluation indicators and output format includes the following steps:

[0089] S131: Based on documents in different fields, text documents containing annotation information are used as input data of the large language model;

[0090] In this embodiment, the annotation information includes named entities, relationships, etc.

[0091] S132: extracting structured information in combination with the evaluation index of the information extraction task;

[0092] In this embodiment, the evaluation indicators of the information extraction task include extraction accuracy evaluation and recall evaluation;

[0093] Among them, the expression for extraction accuracy evaluation is:

[0094] ;

[0095] In the formula, Indicates the extraction accuracy value, that is, the extraction accuracy evaluation result; Indicates the number of correctly extracted information answer texts; The number of texts indicating incorrectly extracted information;

[0096] Among them, the expression of recall evaluation is:

[0097] ;

[0098] In the formula, Represents the recall value, i.e., the recall evaluation result; The number of texts indicating missed information extraction;

[0099] The expression for the score is:

[0100] ;

[0101] In the formula, Represents the harmonic mean based on extraction precision and recall;

[0102] S133: Outputting the extracted structured information list;

[0103] In this embodiment, the output structured information list includes entity names and their attributes.

[0104] Specifically, based on the translation task, the process of predefining the corresponding input, evaluation indicators and output format includes the following steps:

[0105] S141: Using the original sentence as input data for the large language model;

[0106] In this embodiment, the original sentences are in multiple languages.

[0107] S142: Evaluate translation quality based on the evaluation indicators of the translation task;

[0108] Evaluation indicators of translation tasks include translation accuracy assessment and naturalness assessment;

[0109] Among them, the translation accuracy is measured by BLEU score, and the naturalness is measured by perplexity.

[0110] S143: Outputting the translation text in the target language;

[0111] In this embodiment, the translated text in the target language accurately expresses the original meaning and has natural and fluent language expression.

[0112] S2: Receive user query requests and use the Transformer-based classifier model to automatically identify the task type to which the request belongs;

[0113] S21: receiving user query request , and use it as the input of the Transformer model, and after being processed by multiple encoder layers of the Transformer model, output the contextual semantic information of the user's query request;

[0114] In this embodiment, the Transformer model consists of multiple encoder layers; each encoder layer includes a multi-head self-attention mechanism and a feedforward neural network;

[0115] Among them, the expression of a single encoder layer is:

[0116] ;

[0117] In the formula, Indicates The output of the encoding layer; Indicates The output of the encoding layer; represents the normalization function; , represents the total number of encoder layers; when When Indicates user query request ;

[0118] Among them, the expression of the multi-head self-attention mechanism is:

[0119] ;

[0120] In the formula, represents the attention mechanism function; Indicates a query; Indicates the key; represents the value matrix; represents the attention score function; Represents the dimension of the key; represents the transpose of a key;

[0121] Among them, after being processed by all encoder layers, the output of the Transformer model is expressed as:

[0122] ;

[0123] In the formula, Represents the output of the Transformer model, which includes the contextual semantic information of the user query; Indicates The output of the encoder layer.

[0124] S22: using the acquired contextual semantic information as the input of the classifier layer, and outputting the probability distribution of each task category after being processed by the classifier layer;

[0125] In this embodiment, the classifier layer includes a fully connected layer and a softmax layer.

[0126] Among them, the expression of the fully connected layer is:

[0127] ;

[0128] In the formula, Represents the output result of the fully connected layer; Represents the weight matrix of the fully connected layer; Represents the bias vector of the fully connected layer;

[0129] Among them, the softmax layer converts the output results of the fully connected layer into a probability distribution, which is used to represent the probability of each task category. Its expression is:

[0130] ;

[0131] In the formula, Indicates based on user query request , each task category probability.

[0132] S23: based on the probability distribution of each task category, select the task category with the highest probability as the classification result to represent the most relevant task type;

[0133] S3: Based on the identified task type, dynamically select a large language model that is suitable for the field for modeling, and build a large language model evaluation standard based on the corpus data of the corresponding task field;

[0134] S31: extracting corpus text data associated with the identified task type based on the corpus;

[0135] S32: Using the extracted corpus text data as input of the large language model, obtaining the evaluation score of each corpus text data, calculating the comprehensive performance score under the corresponding task type, and completing the construction of the large language model evaluation standard;

[0136] Among them, the expression of the comprehensive performance score under the corresponding task type is:

[0137] ;

[0138] In the formula, Indicates the comprehensive score of the task; Indicates Evaluation scores of text data; , Indicates the total amount of text data.

[0139] S4: Based on the constructed large language model evaluation criteria, multiple candidate large language models are automatically tested and quantitatively scored, and the candidate large language models are screened and weighted based on the scoring results to construct the optimal large language model combination;

[0140] S41: Select multiple candidate large language models;

[0141] S42: Based on the same test template and corpus text data, input them into multiple candidate large language models for automated testing and quantitative scoring;

[0142] S43: Based on the quantitative scoring results, sort and screen the candidate large language models and select the top model with the highest score. A large language model;

[0143] Among them, the highest scoring The selection formula for selecting a large language model is:

[0144] ;

[0145] In the formula, Indicates the top scores among the selected A large language model; Indicates Test score results of a large language model;

[0146] S44: Based on the screened A large language model is used to set the model weights based on the current task performance to build the optimal large language model combination.

[0147] The expression of model weight is:

[0148] ;

[0149] In the formula, Indicates The model weights of a large language model.

[0150] S5: Take the user query request as the input of the optimal large language model combination, perform weighted fusion processing on the output probability of each model in the optimal large language model combination, and use autoregressive generation technology to sample word by word to generate the final text output.

[0151] S51: User query request As the selected The input of a large language model generates the corresponding output probability distribution ;

[0152] S52: Output probability distribution based on major language models , combined with the corresponding model weights Perform weighted fusion processing to output the output probability of the optimal large language model combination ;

[0153] Among them, the output probability expression of the optimal large prediction model combination is:

[0154] ;

[0155] In the formula, Indicates based on user query request , Generate words The output probability of the optimal large prediction model combination; Indicates based on user query request , Generate words Next The output probability of a large language model.

[0156] S53: Use autoregressive generation technology to generate word by word, and sample the output probability of the optimal large language model combination to generate the final text output.

[0157] In this embodiment, when the autoregressive generation technology is used to generate words one by one, the words generated in each step are used as input to continue to generate the next word until a complete sentence or paragraph is output. Specifically, the following steps are included:

[0158] S531: Set the generated text sequence to = , , Indicates the length of the generated text;

[0159] S532: Based on the output probability of the optimal large language model combination, the autoregressive generation technology is used to sample each word to generate the final text output;

[0160] The sampling process expression is:

[0161] ;

[0162] In the formula, Indicates The word sequence generated by step; Represents the previously generated word sequence; Indicates based on user query request The generated word sequence The sampling probability of .

[0163] Example 2

[0164] Based on the same design concept, Figure 2 As shown, this embodiment provides a task-oriented large language model selection and construction system, including a task domain setting module, a task type identification module, a model evaluation standard construction module, a model screening and construction module, and a model output module.

[0165] Specifically, the task domain setting module is used to preset multiple large language model task domains and predefine corresponding input, evaluation indicators and output formats for different large language model task domains;

[0166] Specifically, the task type identification module is used to receive user query requests and automatically identify the task type to which the request belongs using a Transformer-based classifier model;

[0167] Specifically, the model evaluation standard construction module is used to dynamically select a large language model adapted to the field for modeling based on the identified task type, and to build a large language model evaluation standard in combination with the corpus data of the corresponding task field;

[0168] Specifically, the model screening and construction module is used to automatically test and quantitatively score multiple candidate large language models based on the constructed large language model evaluation criteria, and screen and weight the candidate large language models based on the scoring results to build the optimal large language model combination;

[0169] Specifically, the model output module is used to take the user query request as the input of the optimal large language model combination, perform weighted fusion processing on the output probability of each model in the optimal large language model combination, and use autoregressive generation technology to sample word by word to generate the final text output.

[0170] Example 3

[0171] Based on the same technical concept, the embodiment of the present application also provides a computer device, including a memory 1 and a processor 2, such as Figure 3 As shown, the memory 1 stores a computer program, and the processor 2 implements any of the above methods when executing the computer program.

[0172] Among them, the memory 1 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 1 can be an internal storage unit of the task-oriented large language model selection and construction system, such as a hard disk. In other embodiments, the memory 1 can also be an external storage device of the task-oriented large language model selection and construction system, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 1 can also include both an internal storage unit of the task-oriented large language model selection and construction system and an external storage device. The memory 1 can not only be used to store application software and various types of data installed in the task-oriented large language model selection and construction system, such as the code of the task-oriented large language model selection and construction system program, but also can be used to temporarily store data that has been output or is to be output.

[0173] In some embodiments, the processor 2 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip, which is used to run the program code stored in the memory 1 or process data, such as executing the selection and construction system program of a task-oriented large language model.

[0174] The disclosed embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to perform the steps of the method described in the above method embodiment. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0175] The computer program product of the application page content refresh method provided in the disclosed embodiment of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the steps of the method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0176] The disclosed embodiment of the present invention further provides a computer program, which implements any one of the methods of the aforementioned embodiments when executed by a processor. The computer program product can be implemented in hardware, software, or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium, and in another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0177] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0178] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0179] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0180] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0181] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0182] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0183] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0184] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0185] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A task-oriented large language model selection and construction method, It is characterized in that The steps include: Preset multiple large language model task fields, and predefine corresponding input, evaluation indicators and output formats for different large language model task fields; Receive user query requests and use the Transformer-based classifier model to automatically identify the task type to which the request belongs; Based on the identified task type, dynamically select a large language model that is suitable for the field for modeling, and build a large language model evaluation standard based on the corpus data of the corresponding task field; Based on the constructed large language model evaluation criteria, multiple candidate large language models are automatically tested and quantitatively scored, and the candidate large language models are screened and weighted based on the scoring results to construct the optimal large language model combination; The user query request is used as the input of the optimal large language model combination, the output probability of each model in the optimal large language model combination is weighted and fused, and the final text output is generated by sampling word by word using the autoregressive generation technology; The large language model task areas include text generation tasks, semantic understanding tasks, information extraction tasks, and translation tasks; Based on the identified task type, a large language model adapted to the field is dynamically selected for modeling, and the process of building a large language model evaluation standard is specifically composed of the following steps in combination with the corpus data of the corresponding task field: Based on the corpus, extracting corpus text data associated with the identified task type; The extracted corpus text data is used as the input of the large language model, and the evaluation score of each corpus text data is obtained, and the comprehensive performance score under the corresponding task type is calculated to complete the construction of the large language model evaluation standard; Based on the constructed large language model evaluation criteria, multiple candidate large language models are automatically tested and quantitatively scored, and the candidate large language models are screened and weighted based on the scoring results. The process of constructing the optimal large language model combination specifically includes the following steps: Select multiple candidate large language models; Based on the same test template and corpus text data, they are input into multiple candidate large language models for automated testing and quantitative scoring; Based on the quantitative scoring results, the candidate large language models are sorted and screened; Based on the selected large language models, the model weights are set in combination with the current task performance to build the optimal large language model combination.

2. According to the method for selecting and constructing a task-oriented large language model according to claim 1, It is characterized in that The process of presetting multiple large language model task domains and predefining corresponding input, evaluation indicators and output formats for different large language model task domains includes the following steps: Based on the text generation task, a group of unfinished sentences or paragraphs are used as input data of the large language model, and text generation is performed in combination with the evaluation indicators of the text generation task, and a completed text paragraph is output; the evaluation indicators of the text generation task include fluency evaluation, coherence evaluation and accuracy evaluation; Based on the semantic understanding task, two text segments to be compared are used as input data of the large language model, and semantic similarity is judged in combination with the evaluation index of the semantic understanding task, and a similarity score or a matching label is output; the evaluation index of the semantic understanding task includes semantic matching evaluation and accuracy evaluation; Based on the information extraction task, a text document containing annotated information is used as input data of a large language model, structured information is extracted in combination with evaluation indicators of the information extraction task, and a list of extracted structured information is output; the evaluation indicators of the information extraction task include extraction accuracy evaluation and recall rate evaluation; Based on the translation task, the original sentence is used as the input data of the large language model, and the translation quality is evaluated in combination with the evaluation indicators of the translation task, and the translation text in the target language is output; the evaluation indicators of the translation task include translation accuracy evaluation and naturalness evaluation; The fluency evaluation and the naturalness evaluation are measured by perplexity; the coherence evaluation and the translation accuracy evaluation are measured by BLEU score; and the accuracy evaluation is measured by ROUGE-L indicator.

3. According to the method for selecting and constructing a task-oriented large language model according to claim 1, It is characterized in that The process of receiving a user query request and automatically identifying the task type to which the request belongs using a Transformer-based classifier model specifically includes the following steps: Receive user query requests and use them as input to the Transformer model. After being processed by multiple encoder layers of the Transformer model, the contextual semantic information of the user query request is output. The obtained contextual semantic information is used as the input of the classifier layer, and the classifier layer processes and outputs the probability distribution of each task category; Based on the probability distribution of each task category, the task category with the highest probability is selected as the most relevant task type.

4. According to the method for selecting and constructing a task-oriented large language model according to claim 1, It is characterized in that The user query request is used as the input of the optimal large language model combination, the output probability of each model in the optimal large language model combination is weighted and fused, and the process of using autoregressive generation technology to sample word by word to generate the final text output specifically includes the following steps: The user query request is used as the input of each of the selected language models to generate the corresponding output probability distribution; Based on the output probability distribution of each language model, the corresponding model weights are combined to perform weighted fusion processing to output the output probability of the optimal large language model combination; The autoregressive generation technology is used to generate each word, and the output probability of the optimal large language model combination is sampled to generate the final text output.

5. A system for the task-oriented large language model selection and construction method according to claim 1, It is characterized in that include: A task domain setting module, which is used to preset multiple large language model task domains and predefine corresponding input, evaluation indicators and output formats for different large language model task domains; The task type identification module is used to receive user query requests and automatically identify the task type to which the request belongs using a Transformer-based classifier model; A model evaluation standard construction module is used to dynamically select a large language model adapted to the field for modeling based on the identified task type, and to construct a large language model evaluation standard in combination with the corpus data of the corresponding task field; The model screening and construction module is used to automatically test and quantitatively score multiple candidate large language models based on the constructed large language model evaluation criteria, and to screen and weight the candidate large language models based on the scoring results to construct the optimal large language model combination; The model output module is used to take the user query request as the input of the optimal large language model combination, perform weighted fusion processing on the output probability of each model in the optimal large language model combination, and use autoregressive generation technology to sample word by word to generate the final text output.

6. An electronic device, It is characterized in that The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium storing a computer program, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images