Model cross-validation disambiguation selection model training method, selection method and device

Abstract

The application provides a multi-large model cross-validation disambiguation selection model training method, selection method and device. A strategy network and an evaluation network are constructed based on reinforcement learning, a plurality of large language models are cross-validated, a reward model pre-trained based on human preference data is introduced to estimate a reward value, so that the selection result of the plurality of large language models in the reinforcement learning process is more in line with the preference and expectation of humans. The decision network introduces state parameters including a question type and a large language model historical performance statistical quantity, can adapt to different question scenarios, shows good scalability and flexibility, comprehensively evaluates the performance of each large language model, and can make more accurate selection decisions.

Description

Technical Field

[0001] This invention relates to the field of computer data processing technology, and in particular to a multi-model cross-validation disambiguation model training method, selection method, and apparatus. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) have become a hot topic in the field of natural language processing (NLP). These models, trained through deep learning and massive amounts of data, are able to understand and generate natural language and are widely used in various fields such as text classification, machine translation, and question answering systems. However, due to the complexity and diversity of languages, these models often face challenges in handling ambiguity.

[0003] Ambiguity refers to the possibility that the same input may correspond to multiple reasonable but different interpretations or answers. For example, a simple question, "How is the phone?", could refer to multiple aspects such as the phone's quality, design, and price. Different large models may provide different answers based on their training data and algorithm characteristics. Existing technologies include methods for disambiguating author names based on graph convolutional semi-supervised learning. These methods leverage the advantages of graph convolutional neural networks in graph semi-supervised learning by constructing a network of collaboration and institutional relationships between papers, combined with the semantic representation vectors of the papers, to achieve author name disambiguation. This method has achieved good results in handling author name disambiguation, but it primarily targets author identification rather than disambiguation of the model output.

[0004] Therefore, a solution is urgently needed to find a way to select or synthesize the most credible answer from multiple models. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a multi-model cross-validation disambiguation selection model training method, selection method and apparatus to eliminate or improve one or more defects in the prior art, so as to solve the problem that the prior art cannot perform disambiguation selection for the answers output by multiple large language models.

[0006] One aspect of the present invention provides a method for cross-validation disambiguation selection of model training for multi-model systems, the method comprising the following steps:

[0007] Multiple large language models process multiple pre-set question data and obtain corresponding answers, and construct an answer set for each question data;

[0008] Construct a state space, which includes a vector representation of the question data, a question type encoding, an identifier vector of the large language model, historical performance statistics of the large language model, a set of vector representations of each answer corresponding to the question data, a similarity matrix between the answers, and a set of answer satisfaction vectors based on preset link feedback.

[0009] Construct an action space, which includes vectors for answer selection given by various language models;

[0010] Based on reinforcement learning, a policy network is obtained, which takes as input the vector representation of the question data in the state space, the question type encoding, the identifier vector of the large language model, the historical performance statistics of the large language model, the vector representation set of each answer corresponding to the question data, and the similarity matrix between the answers, and outputs an answer selection prediction vector. An evaluation network is obtained, which takes as input the parameters of the state space and the answer selection prediction vector and outputs a value estimate. Based on a pre-trained reward model, the answer corresponding to the question data is taken as input and the preference value of each answer is output as a reward value. The mean squared error of the value estimate and the reward value is calculated, and a truncated target loss is introduced. A loss function is calculated, and in multiple iterations, the parameters of the policy network and the evaluation network are updated by minimizing the loss function. The final policy network is used as a cross-validation disambiguation selection model.

[0011] In some embodiments, the vector representation of the question data and the corresponding vector representation of each answer are obtained by processing using the bag-of-words model, TF-IDF model, Word2Vec model, GloVe model or FastText model.

[0012] In some embodiments, the question type encoding uses a one-hot vector for encoding and labeling. The question types include information query, operation guide, product consultation, technical support, health and medical care, education and learning, entertainment and leisure, shopping and consumption, laws and regulations, emotional and psychological issues, and career and employment issues.

[0013] The identifier vector of the large language model is encoded and marked using one-hot vectors;

[0014] The historical performance statistics of the large language model include precision, recall, and / or F1 score;

[0015] In the similarity matrix between the answers, cosine similarity or Euclidean distance is used to calculate the similarity between the vector representations of each answer.

[0016] In some embodiments, the policy network and the evaluation network employ a multilayer perceptron, a convolutional neural network, or a recurrent neural network.

[0017] In some embodiments, the pre-training step of the reward model includes:

[0018] Obtain a training sample set containing multiple samples, each sample containing question data and a vector representation of its answer, and labeling it with user preference values ​​as tags;

[0019] Obtain an initial neural network model, which includes sequentially connected convolutional layers and fully connected layers; output the vector representation of the question data and its answers in the sample, and output a predicted value for the user preference value;

[0020] The initial neural network is trained using the training sample set. A loss function is constructed using the mean squared deviation of the predicted value and the label. The parameters of the initial neural network model are updated by minimizing the loss function to obtain the reward model.

[0021] In some embodiments, the mean squared error between the value estimate and the reward value is calculated, and a truncated target loss is introduced to calculate the loss function, including:

[0022] The value function regression loss is calculated using the following formula:

[0023]

[0024] Where N represents the number of samples, s t Indicates the state at time t, a t This represents the action chosen at time t. R represents the value estimate. t Indicates the reward value;

[0025] The loss of the truncated target is calculated using the following formula:

[0026]

[0027] in, For the current strategy In state s t Next, execute action a t The probability of; For the old strategy In state s t Next, execute action a t The probability of; Importance sampling ratio; Let be the dominance function, representing action a. t Compared to the performance of the current strategy; clip is a clipping function that restricts the importance sampling ratio to [1-∈, 1+∈];

[0028] The formula for calculating the loss function is:

[0029]

[0030] Where λ is the weighting coefficient.

[0031] On the other hand, the present invention also provides a method for disambiguation selection in multi-model cross-validation, comprising:

[0032] Input the target question data into multiple pre-set large language models and obtain the corresponding answer dataset;

[0033] Calculate the similarity between each answer in the answer dataset and construct a similarity matrix;

[0034] Obtain the vector representation of the question data, the question type encoding, the identifier vector of the large language model, the historical performance statistics of the large language model, the vector representation set of each answer corresponding to the question data, and the similarity matrix. Input the cross-validation disambiguation selection model in the cross-validation disambiguation selection model training method for multiple models mentioned above, and output the selection result of the answer data.

[0035] On the other hand, the present invention also provides a multi-model cross-validation disambiguation device, including a processor, a memory, and a computer program / instructions stored in the memory, wherein the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the above method.

[0036] On the other hand, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0037] On the other hand, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0038] The beneficial effects of the present invention are at least as follows:

[0039] The multi-model cross-validation disambiguation selection model training method, selection method, and apparatus described in this invention constructs a policy network and an evaluation network based on reinforcement learning, performs cross-validation on multiple large language models, and introduces a reward model pre-trained based on human preference data to estimate reward values. This ensures that the selection results of the output answers from multiple large language models during reinforcement learning better align with human preferences and expectations. The decision network incorporates state parameters including question type and historical performance statistics of the large language models, enabling it to adapt to different problem scenarios, demonstrating good scalability and flexibility. By comprehensively evaluating the performance of various language models, it can make more accurate selection decisions.

[0040] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0041] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0042] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings:

[0043] Figure 1 This is a flowchart illustrating the multi-model cross-validation disambiguation selection model training method according to an embodiment of the present invention.

[0044] Figure 2 This is a logical schematic diagram of a multi-model cross-validation disambiguation selection model training method according to an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0046] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0047] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0048] To overcome the ambiguity of a single large language model in the question-answering process, this invention utilizes multiple large language models for cross-validation, introduces reinforcement learning to construct a policy network that selects from the answers provided by multiple large language models, and introduces a pre-trained reward model to fit human users' preferences for answers as the reward value. This enables the policy network optimized by reinforcement learning to select answers that better match human users' preferences and expectations, thus eliminating the ambiguity that may exist in a single large language model.

[0049] Specifically, this invention provides a method for cross-validation disambiguation selection of model training for various model sizes, such as... Figure 1 and Figure 2 As shown, the method includes the following steps S101 to S104:

[0050] Step S101: Multiple large language models process multiple preset question data and obtain corresponding answers, and construct the answer set for each question data.

[0051] Step S102: Construct the state space, which includes the vector representation of the question data, the question type encoding, the identifier vector of the large language model, the historical performance statistics of the large language model, the vector representation set of each answer corresponding to the question data, the similarity matrix between answers, and the set vector of answer satisfaction based on the preset link feedback.

[0052] Step S103: Construct the action space, which includes vectors for the answer selection given by various language models.

[0053] Step S104: Based on reinforcement learning, obtain the policy network with the vector representation of the question data in the state space, the question type encoding, the identifier vector of the large language model, the historical performance statistics of the large language model, the vector representation set of each answer corresponding to the question data, and the similarity matrix between the answers as input, and output the answer selection prediction vector; obtain the evaluation network with the parameters of the state space and the answer selection prediction vector as input and output the value estimate; based on the pre-trained reward model, with the answer corresponding to the question data as input and output the preference value of each answer as the reward value; calculate the mean square error of the value estimate and the reward value, and introduce the truncated target loss, calculate the loss function, and update the parameters of the policy network and the evaluation network by minimizing the loss function in multiple iterations, and use the final policy network as the cross-validation disambiguation selection model.

[0054] In step S101, the large language model applicable to this invention can include multiple categories. Depending on the application scenario, the GPT series (Generative Pre-trained Transformer), developed by OpenAI, can be selected, including GPT-1, GPT-2, GPT-3, and their variants such as GPT-3.5 and GPT-4. These models perform well in various natural language processing tasks. BERT (Bidirectional Encoder Representations from Transformers), developed by Google, is a pre-trained language representation model primarily used for understanding the meaning of words in context. T5 (Text-to-Text Transform), developed by Google, is a unified text-to-text conversion model capable of handling various NLP tasks, such as translation, summarization, and question answering. RoBERTa (A Robustly Optimized BERT Pretraining Approach), an improved version of BERT proposed by Facebook AI Research (FAIR), improves performance through dynamic masking and longer training time. XLNet, jointly proposed by CMU and Google, is an improved Transformer model. Megatron-LM, jointly developed by NVIDIA and Microsoft, is a large-scale language model focused on efficient training and inference. Turing-NLG, developed by Microsoft, is a large-scale generative pre-trained model that has since been integrated into multiple Microsoft products and services.

[0055] During the application process, multiple large language models are selected to answer the same question simultaneously, resulting in a question set.

[0056] In steps S102 and S103, to introduce reinforcement learning to select answers from multiple large language models that better align with human preferences, cross-validation is used to eliminate ambiguity in individual large language models, and a state space and action space are constructed as preparation. The state space incorporates vector representations of the question data, question type encoding, identifier vectors of the large language models, historical performance statistics of the large language models, a set of vector representations of each answer corresponding to the question data, a similarity matrix between answers, and a set of answer satisfaction vectors based on pre-defined feedback links. The action space represents the actions for selecting answers from the outputs of the various language models.

[0057] In some embodiments, the vector representation of the question data and the corresponding vector representation of each answer are obtained by processing the data using the bag-of-words model (Bow), TF-IDF model, Word2Vec model, GloVe model, or FastText model.

[0058] The bag-of-words model is one of the simplest text representation methods. It decomposes text into individual words and constructs a vocabulary that records the frequency of each word in the text. In processing, the problem data is first segmented and stop word removed, and then the text is encoded based on the vocabulary to generate a high-dimensional sparse vector. TF-IDF (Term Frequency-Inverse Document Frequency) is an improvement on BoW, considering not only term frequencies but also the importance of words in the entire corpus. Its processing steps include: first, calculating the term frequency (TF) and inverse document frequency (IDF) of each word, and then multiplying them to obtain the weight of each word, generating a vectorized representation. Word2Vec is a neural network-based word embedding method that maps words to a low-dimensional continuous vector space. In processing, the text needs to be preprocessed (e.g., segmented), and then the corpus is trained using a pre-trained model or a self-built model (such as CBOW or Skip-Gram) to obtain the semantic vector representation of each word. GloVe (Global Vectors for Word Representation) generates word vectors by combining matrix factorization with co-occurrence information of words in a global corpus. Its processing steps are similar to Word2Vec, but it places greater emphasis on global statistical characteristics. FastText is an improved version of Word2Vec; it not only considers the words themselves but also captures more sub-word information by decomposing words into character n-grams. In this way, FastText is more suitable for processing out-of-vocabulary words or misspelled text. In practice, depending on the task requirements, an appropriate vectorization method is selected to process the question data and its corresponding answers to construct feature representations suitable for subsequent model training.

[0059] In some embodiments, question type encoding uses one-hot vectors for labeling. Question types include information query, operation guide, product consultation, technical support, health and medical, education and learning, entertainment and leisure, shopping and consumption, legal and regulatory, emotional and psychological, and career and employment. By further classifying question types, it is possible to learn more precisely the differences in human users' preferences for answers to different types of questions.

[0060] The identifier vector of the large language model is encoded and labeled using one-hot vectors.

[0061] Specifically, a one-hot vector is a sparse vector where only one position has a value of 1, and the rest are 0, used to identify a specific category. Suppose there are N different problem types, such as "calculation," "logic reasoning," and "text analysis." First, each type is assigned a unique index number (e.g., 0 for "calculation," 1 for "logic reasoning," and so on). Then, one-hot vectors are constructed based on these indices. For example, if there are four types, and "logic reasoning" has an index of 1, the corresponding one-hot vector would be [0,1,0,0][0,1,0,0][0,1,0,0]. This encoding method clearly distinguishes different problem types and does not introduce additional size relationships into the model.

[0062] The method for identifying large language models (such as GPT-3, BERT, etc.) is similar. Assuming there are M different models to distinguish, each model is also assigned a unique number. For example, GPT-3 is numbered 0, BERT is numbered 1, and RoBERTa is numbered 2. If BERT is used in the current task, its corresponding one-hot vector is [0,1,0][0,1,0][0,1,0].

[0063] Specifically, a state can be represented as a vector:

[0064] s = [q vec ,q cat ,m ids ,m stats ,a vecs ,sim matrix ,h feedbacks ];

[0065] Where, q vec It is a vector representation of the problem data, q cat It is the question type encoding, m ids It is the identifier vector of the pre-trained large language model, m stats It is the historical performance statistics of a pre-trained large language model, a vecs It is the set of vector representations of each answer, sim matrix It is the similarity matrix between the answers, h feedbacks It is a vector representing the set of human satisfaction with the answers.

[0066] Similarly, in the action space, the answer selection vector can also be represented by one-hot encoding. Since the action mainly involves selecting from multiple pre-trained large models' answers, the action space can be defined as:

[0067] A = {a1, a2, ..., a} n}

[0068] Where n is the number of pre-trained large models participating in providing the answer, and a i This represents the action of selecting the answer provided by the i-th pre-trained large model as the final output. For example, if there are 3 pre-trained large models providing answers, then the action space is {a1, a2, a3}, corresponding to selecting the answers from the first, second, and third models, respectively. a1, a2, and a3 are marked with 0 or 1 to indicate whether they are selected.

[0069] Historical performance statistics for large language models include precision, recall, and / or F1 score;

[0070] In the similarity matrix between answers, cosine similarity or Euclidean distance is used to calculate the similarity between the vector representations of each answer.

[0071] In step S104, the policy network (Actor) directly generates the agent's behavioral policy, i.e., choosing the best action given a state. It outputs actions in the form of a probability distribution, thus guiding the agent's actions in the environment. The evaluation network (Critic) estimates the "goodness" or "badness" of the agent taking a certain action, i.e., evaluates the value of the action or the value of the state. The policy network relies on the feedback (value function) provided by the evaluation network to improve its policy; the evaluation network continuously learns more accurate value estimates through the agent's behavior and reward signals. This interactive mechanism accelerates policy optimization and improves the agent's learning efficiency and stability.

[0072] In some embodiments, the policy network and evaluation network employ a multilayer perceptron, a convolutional neural network, or a recurrent neural network.

[0073] In some embodiments, the pre-training steps of the reward model (RLHF) include steps S201 to S203:

[0074] Step S201: Obtain a training sample set containing multiple samples, each sample containing question data and its answer vector representation, and labeling user preference values ​​as labels.

[0075] Step S202: Obtain the initial neural network model, which includes sequentially connected convolutional layers and fully connected layers; output the vector representation of the question data and its answer in the sample, and output the estimated value for the user's preference value.

[0076] Step S203: Train the initial neural network using the training sample set, construct a loss function using the mean squared deviation of the predicted value and the label, minimize the loss function to update the parameters of the initial neural network model, and obtain the reward model.

[0077] In steps S201-S203, each sample includes the question data and its answer vector representation, obtained through previous text vectorization methods (such as TF-IDF, Word2Vec, etc.). Additionally, the sample needs to include user preference values ​​as labels, which can be annotated using user history, ratings, or interaction records for supervised training. A neural network model is initialized, with a basic structure including convolutional layers and fully connected layers for feature extraction and pattern learning, respectively. The model is input with the question data and the answer vector representation, and outputs a predicted value for the user preference value, i.e., a prediction of the user's preference for the question-answer combination. The neural network is trained using a training sample set. During training, the model constructs a loss function by calculating the mean squared error (MSE) between the predicted value and the actual user preference value (label). Through backpropagation and optimization algorithms (such as SGD or Adam), the model parameters are continuously adjusted to minimize the loss function, ultimately obtaining an optimized reward model. This model can be used to predict user preferences for new question-answer pairs, thereby assisting system decision-making.

[0078] Reward model output: R i =v head (h i ), where h i R represents the input to the initial neural network model. i This represents the output of the initial neural network model.

[0079] The loss function for training the reward model is constructed as follows: Where, r i B represents the truth value corresponding to the label, and B represents the number of samples.

[0080] In some embodiments, the mean squared error of the value estimate and reward value is calculated, and a truncated target loss is introduced to calculate the loss function, including:

[0081] The value function regression loss is calculated using the following formula:

[0082]

[0083] Where N represents the number of samples, s t Indicates the state at time t, a t This represents the action chosen at time t. R represents the value estimate. t This represents the reward value.

[0084] The loss of the truncated target is calculated using the following formula:

[0085]

[0086] in, For the current strategy In state s t Next, execute action a t The probability of; For the old strategy In state s t Next, execute action a t The probability of; Importance sampling ratio; Let be the dominance function, representing action a. t The performance relative to the current strategy; clip is a clipping function that restricts the importance sampling ratio to [1-∈, 1+∈].

[0087] The formula for calculating the loss function is:

[0088]

[0089] Where λ is the weighting coefficient.

[0090] On the other hand, the present invention also provides a method for disambiguation selection of multiple model cross-validation, including steps S301 to S303:

[0091] Step S301: Input the target question data into multiple preset large language models and obtain the corresponding answer dataset.

[0092] Step S302: Calculate the similarity between answers in the answer dataset and construct a similarity matrix.

[0093] Step S303: Obtain the vector representation of the question data, the question type encoding, the identifier vector of the large language model, the historical performance statistics of the large language model, the vector representation set of each answer corresponding to the question data, and the similarity matrix. Input the cross-validation disambiguation selection model in the cross-validation disambiguation selection model training method for multiple models described in steps S101 to S104, and output the selection result of the answer data.

[0094] On the other hand, the present invention also provides a multi-model cross-validation disambiguation device, including a processor, a memory, and a computer program / instructions stored in the memory, wherein the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the above method.

[0095] On the other hand, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0096] On the other hand, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0097] The present invention has the following advantages over the prior art:

[0098] (1) Improve the accuracy and reliability of answers:

[0099] By cross-validating multiple large models and combining confidence matrices and RLHF, this invention can reduce the bias that a single model might produce, improving the consistency and accuracy of the answers. This method not only improves the accuracy of the answers but also provides uncertainty quantification and reliability guarantees for model predictions.

[0100] (2) Optimize the use of computing resources:

[0101] This invention designs an efficient cross-validation process that reduces computational resource consumption, making cross-validation of large models more practical. This solves the problem of high computational cost in traditional cross-validation methods when dealing with large datasets.

[0102] (3) Enhance the flexibility and adaptability of the model:

[0103] This invention develops a machine learning-based disambiguation method that can adapt to different contexts and domains, improving disambiguation performance. This method is more flexible and adaptable than existing rule-based or heuristic methods.

[0104] (4) Incorporate human feedback to optimize the model:

[0105] Through RLHF, this invention enables model optimization based on human feedback, further improving the quality of the model's output. This method allows the model to learn how to better meet the expectations and preferences of human users.

[0106] (5) Improve the model's generalization ability:

[0107] This invention improves the generalization ability of a model across different datasets and application scenarios by combining cross-validation and RLHF. This addresses the problem that a single model may be too specific to its training data.

[0108] Based on the above advantages, this invention demonstrates significant improvements and innovations in technical solutions, model optimization, result interpretation, security, ethics, generalization ability, confidence matrix construction, reinforcement learning strategy fine-tuning, and uncertainty quantification.

[0109] Corresponding to the above method, the present invention also provides an apparatus / system including a computer device, the computer device including a processor and a memory, the memory storing computer instructions, the processor executing the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the apparatus / system performs the steps of the method as described above.

[0110] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0111] In summary, the multi-model cross-validation disambiguation selection model training method, selection method, and apparatus described in this invention constructs a policy network and an evaluation network based on reinforcement learning, performs cross-validation on multiple large language models, and introduces a reward model pre-trained based on human preference data to estimate reward values. This ensures that the selection results of the output answers from multiple large language models during reinforcement learning better align with human preferences and expectations. The decision network incorporates state parameters including question type and historical performance statistics of the large language models, enabling it to adapt to different problem scenarios, demonstrating good scalability and flexibility. By comprehensively evaluating the performance of various language models, it can make more accurate selection decisions.

[0112] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0113] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0114] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0115] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for cross-validation disambiguation selection of model training for multi-scale models, characterized in that, The method includes the following steps: Multiple large language models process multiple pre-set question data and obtain corresponding answers, and construct an answer set for each question data; Construct a state space, which includes a vector representation of the question data, a question type encoding, an identifier vector of the large language model, historical performance statistics of the large language model, a set of vector representations of each answer corresponding to the question data, a similarity matrix between the answers, and a set of answer satisfaction vectors based on preset link feedback. Construct an action space, which includes vectors for answer selection given by various language models; Based on reinforcement learning, a policy network is obtained, which takes as input the vector representation of the question data in the state space, the question type encoding, the identifier vector of the large language model, the historical performance statistics of the large language model, the vector representation set of each answer corresponding to the question data, and the similarity matrix between the answers, and outputs an answer selection prediction vector. An evaluation network is obtained, which takes as input the parameters of the state space and the answer selection prediction vector and outputs a value estimate. Based on a pre-trained reward model, the answer corresponding to the question data is taken as input and the preference value of each answer is output as a reward value. The mean squared error of the value estimate and the reward value is calculated, and a truncated target loss is introduced. A loss function is calculated, and in multiple iterations, the parameters of the policy network and the evaluation network are updated by minimizing the loss function. The final policy network is used as a cross-validation disambiguation selection model.

2. The method for cross-validation disambiguation selection of model training for multiple models according to claim 1, characterized in that, The vector representations of the question data and the corresponding vector representations of each answer are obtained by processing using the bag-of-words model, TF-IDF model, Word2Vec model, GloVe model, or FastText model.

3. The method for cross-validation disambiguation selection of model training for multiple models according to claim 1, characterized in that, The question type encoding uses one-hot vectors for encoding and labeling. The question types include information query, operation guide, product consultation, technical support, health and medical care, education and learning, entertainment and leisure, shopping and consumption, laws and regulations, emotional and psychological issues, and career and employment issues. The identifier vector of the large language model is encoded and marked using one-hot vectors; The historical performance statistics of the large language model include precision, recall, and / or F1 score; In the similarity matrix between the answers, cosine similarity or Euclidean distance is used to calculate the similarity between the vector representations of each answer.

4. The method for cross-validation disambiguation selection of model training for multiple models according to claim 1, characterized in that, The policy network and the evaluation network employ a multilayer perceptron, a convolutional neural network, or a recurrent neural network.

5. The method for cross-validation disambiguation selection of model training for multiple models according to claim 1, characterized in that, The pre-training steps of the reward model include: Obtain a training sample set containing multiple samples, each sample containing question data and a vector representation of its answer, and labeling it with user preference values ​​as tags; Obtain an initial neural network model, which includes sequentially connected convolutional layers and fully connected layers; output the vector representation of the question data and its answers in the sample, and output a predicted value for the user preference value; The initial neural network is trained using the training sample set. A loss function is constructed using the mean squared deviation of the predicted value and the label. The parameters of the initial neural network model are updated by minimizing the loss function to obtain the reward model.

6. The method for cross-validation disambiguation selection of model training for multiple models according to claim 1, characterized in that, Calculate the mean squared error between the value estimate and the reward value, and introduce a truncated target loss to calculate the loss function, including: The value function regression loss is calculated using the following formula: Where N represents the number of samples, s t Indicates the state at time t, a t This represents the action chosen at time t. R represents the value estimate. t Indicates the reward value; The loss of the truncated target is calculated using the following formula: in, For the current strategy In state s t Next, execute action a t The probability of; For the old strategy In state s t Next, execute action a t The probability of; Importance sampling ratio; Let be the dominance function, representing action a. t The performance relative to the current strategy; clip is a clipping function that restricts the importance sampling ratio to [1-∈, 1+∈]. The formula for calculating the loss function is: Where λ is the weighting coefficient.

7. A method for disambiguation selection in multi-model cross-validation, characterized in that, include: Input the target question data into multiple pre-set large language models and obtain the corresponding answer dataset; Calculate the similarity between each answer in the answer dataset and construct a similarity matrix; Obtain the vector representation of the question data, the question type encoding, the identifier vector of the large language model, the historical performance statistics of the large language model, the vector representation set of each answer corresponding to the question data, and the similarity matrix. Input the cross-validation disambiguation selection model in the cross-validation disambiguation selection model training method for multiple models as described in any one of claims 1 to 6, and output the selection result of the answer data.

8. A multi-model cross-validation disambiguation device, comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

Images