Retrieval question answering model selection method based on large language model prompt and embedding
By combining large language model prompts and embedding methods with prior knowledge and thought chain technology, and using soft-label maximum marginal likelihood scores to evaluate model performance, the problems of high resource consumption and inaccurate evaluation in the model selection process are solved, and efficient and accurate model selection is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIHANG UNIV
- Filing Date
- 2024-05-21
- Publication Date
- 2026-05-29
AI Technical Summary
When faced with a large number of candidate pre-trained language models, existing methods consume a lot of time and computational resources in the model selection process, and model retrieval that relies on expert knowledge lacks reliability and efficiency. Hard label evaluation methods cannot accurately describe the relationship between queries and documents, resulting in inaccurate transferability estimation.
We adopt a method based on large language model prompts and embeddings. By constructing a model to select task scenarios, introducing prior knowledge and thought chain technology, and using dual encoders to generate sentence embeddings, we calculate the maximum marginal likelihood score of soft labels to achieve efficient and accurate model selection.
It shortens model selection time, improves the reliability and accuracy of model retrieval, reduces computational resource consumption, accurately evaluates model performance, and enhances the efficiency and accuracy of model selection.
Smart Images
Figure CN118446250B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of natural language processing, and specifically relates to a retrieval-based question-answering model selection method based on large language model prompts and embedding. Background Technology
[0002] With the widespread adoption of mobile smart devices such as computers and the rapid rise of the internet, informatization has become an irreversible trend. Currently, internet users are located all over the world, and the diverse and convenient ways to access information have made people's lives easier and more efficient. However, this is accompanied by an explosion of information flooding people's daily lives, making the efficient and accurate retrieval of target information from massive amounts of data a pressing issue.
[0003] Question answering, a key task in Natural Language Processing (NLP), addresses the challenge of analyzing and processing user queries or questions, and returning accurate answers in natural language. As a crucial task in NLP, question answering has widespread and common applications in daily life. Commonly used search engines like Baidu and Google are examples of this task, while recently emerging chatbot programs, such as OpenAI's ChatGPT and Bing Chat, which combines the two, possess remarkable understanding capabilities and can communicate with users like humans. It can be said that question answering has always been an important research direction in NLP, and with continuous technological innovation and progress, it has achieved many remarkable results, making research on this task of great significance.
[0004] Based on the method of answer generation, question answering can be divided into generative question answering and retrieval question answering. Generative question answering generates new text as an answer based on the provided question or query. Retrieval question answering, on the other hand, directly retrieves documents related to a given query from a candidate set. Compared to generative question answering, retrieval question answering directly retrieves information related to the query from a predefined document library, greatly improving the efficiency and accuracy of the answer. Therefore, this paper will focus on retrieval question answering. In recent years, pre-trained language models, represented by BERT, have developed rapidly and have been widely used in retrieval question answering due to their excellent performance. However, for the same dataset, the performance of different pre-trained language models may vary significantly. Therefore, how to select the best-performing pre-trained language model from a wide range of candidate models has become an urgent problem to be solved. Summary of the Invention
[0005] To address the aforementioned technical problems, this invention provides a retrieval-based question-answering model selection method based on large language model prompts and embeddings, comprising the following steps:
[0006] Step S1: Prompts for building a large language model, including: the application scenario of the model selection task and the selected target dataset. and A set of candidate pre-trained models ;
[0007] Step S2: For the aforementioned prompt, introduce Examples provide prior knowledge for large language models to enhance their performance in model selection tasks; wherein, the examples include: example datasets. List of example candidate models and example candidate models in The performance ranking list on the platform, among which, The number of candidate models contained in a given example;
[0008] Step S3: Apply the thought chain to the prompts to provide step-by-step guidance for the large language model to complete the model retrieval task. Based on the prompts, the large language model outputs the candidate pre-trained model... Top performance Model retrieval results As the final model retrieval result;
[0009] Step S4: Using two The dual encoder is constructed in query set and document collection The resulting sentence embeddings are encoded as follows: and and in and Based on the sample features ;
[0010] Step S5: Utilize the large language model in The above query set and document collection Encode the sentences to obtain their coded embeddings. and After introducing the whitening mechanism, the whitened sentence embeddings are obtained. and The soft label is obtained by calculating the normalized cosine similarity between the two. ;
[0011] Step S6: Obtain the features of the whitened sample The parameters used are Modeling with marginal likelihood and The compatibility between them is assessed; the maximum marginal likelihood score of the soft label is calculated as the final transferability score of each model in the model ranking stage, and the model with the highest score is selected as the final model selection result.
[0012] Beneficial effects:
[0013] 1. Existing common and direct model selection methods involve brute-force fine-tuning each model and selecting the optimal model based on the final fine-tuning performance. However, this process consumes significant time and computational resources. To address this issue, a more efficient model selection method is transferability estimation, which focuses on quantifying the transferability of models and measuring their fine-tuning performance in a priori manner. Previous methods have only validated the efficiency of transferability estimation on small candidate pools. However, when faced with a large number of candidate models, the time and resource consumption caused by the intensive computational process of transferability estimation, which requires encoding sample features, is not negligible. The model selection method proposed in this invention consists of two stages: first, model retrieval is used to quickly and efficiently narrow down the candidate pool; then, transferability estimation is used to rank the models. Therefore, compared to brute-force fine-tuning and intensive transferability estimation, it is more efficient and accurate.
[0014] 2. For existing model retrieval, researchers often use specialized knowledge to screen models. However, due to knowledge limitations and the significant time required for information searching, relying solely on expert knowledge makes model retrieval unreliable and inefficient. The model retrieval proposed in this invention utilizes unprecedented world knowledge from large language models, efficiently completing tasks through dialogue prompts, thus solving the problems arising from relying solely on expert knowledge. Furthermore, in the model retrieval stage, this invention incorporates context learning technology, providing a real retrieval result from a given dataset within its candidate model pool as an example, thereby providing prior knowledge to the large language model and helping it better understand the task mechanism of model selection. Moreover, the model retrieval stage of this invention also incorporates a thought chain technique, designing a series of intermediate reasoning steps to progressively prompt and guide the large model to output the final model retrieval result, thus improving the performance of large language models in complex reasoning tasks such as model retrieval.
[0015] 3. Current transferability estimation methods typically evaluate the compatibility between sample features, i.e., sentence embeddings in retrieval question answering tasks, and hard labels, i.e., one-hot labels. However, hard labels cannot accurately describe the relationship between queries and documents. For example, some negative documents that are only slightly relevant to the query should be assigned a label value between 0 and 1 instead of 0. Based on this inaccurate data, transferability estimation methods cannot accurately evaluate model performance. To address this issue, one feasible approach is to train a model to obtain soft labels for each sample, thereby correcting the original hard labels. However, training the model is time-consuming. The model ranking method proposed in this invention offers a transferability estimation method based on soft label maximum likelihood. That is, it uses sentence embeddings encoded by a large model to capture the semantics of sentences and generates accurate soft labels based on this. The entire process only requires forward propagation of data samples to the large model, thus saving the time-consuming training process. Then, the maximum marginal likelihood is used to calculate the transferability score of text embeddings to soft labels. Therefore, compared with previous transferability estimation methods, the soft label maximum likelihood used in the model ranking stage of this invention is more efficient and accurate. Attached Figure Description
[0016] Figure 1 This is a schematic diagram of a retrieval-based question-answering model selection method based on large language model prompts and embedding, according to the present invention.
[0017] Figure 2 A schematic diagram illustrating the architecture selection for a retrieval-based question-answering model that utilizes large language model hints and embeddings. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0019] Example 1
[0020] like Figure 1 As shown in the figure, the present invention provides a retrieval-based question-answering model selection method based on large language model prompts and embeddings, which includes the following steps:
[0021] Step S1: Prompts for building a large language model, including: the application scenario of the model selection task and the selected target dataset. and A set of candidate pre-trained models ;
[0022] Step S2: For the prompt, introduce... These examples provide prior knowledge for large language models to enhance their performance in model selection tasks; the examples include: example datasets. List of example candidate models and example candidate models in The performance ranking list on the platform, among which, The number of candidate models contained in a given example;
[0023] Step S3: Apply the thought chain to the prompts to provide step-by-step guidance for the large language model to complete the model retrieval task. Based on the prompts, the large language model outputs candidate pre-trained models. Top performance Model retrieval results As the final model retrieval result;
[0024] Step S4: Using two The dual encoder is constructed in query set and document collection The resulting sentence embeddings are encoded as follows: and and in and Based on the sample features ;
[0025] Step S5: Utilize the large language model in The above query set and document collection Encode the sentences to obtain their embeddings. and After introducing the whitening mechanism, the whitened sentence embeddings are obtained. and The soft label is obtained by calculating the normalized cosine similarity between the two. ;
[0026] Step S6: Obtain the features of the whitened sample The parameters used are Modeling with marginal likelihood and Compatibility between models; calculate the maximum marginal likelihood score of the soft label as the final transferability score of each model in the model ranking stage, and select the model with the highest score as the final model selection result.
[0027] In one embodiment, step S1 above, which provides suggestions for building a large language model, includes: the application scenario of the model selection task and the selected target dataset. and A set of candidate pre-trained models Specifically, it includes:
[0028] The following is an example of a prompt constructed according to an embodiment of the present invention:
[0029] 1.With the growing number of Pre-trained Language Models recently, fine-tuning all
[0030] 2.PLMs for model selection is not practical,I usually don't know which PLM should be
[0031] 3.selected to achieve the best dataset performance. Here is the problem I want you to solve:
[0032] 4.For ReQA BioASQ-9b dataset, and the candidate models:
[0033] 5.[1] bert-base-uncased
[0034] 6.[2] bert-base-cased
[0035] 7.…
[0036] 8.
[50] chkla / roberta-argument
[0037] 9.Please rank all the 50 models. Rember that you don't have to givean accurate ranking list,
[0038] 10.any possible inference is OK.
[0039] Lines 1-4 of the above prompt describe the application scenario for the model selection task, with line 4 specifying the target dataset for the task. This refers to ReQA BioASQ-9b. Each line from lines 5 to 8 lists a candidate pre-trained model. (Line 7 represents the 3rd line not listed) 49 candidate pre-trained models), totaling The set consists of pre-trained models. In this example Lines 9-10 indicate the expected output format of the large language model.
[0040] In one embodiment, step S2 above: For the prompt, introduce These examples provide prior knowledge for large language models to enhance their performance in model selection tasks; the examples include: example datasets. List of example candidate models and example candidate models in The performance ranking list on the platform, among which The number of example candidate models contained in an example, according to the present invention The model in The minor adjustments made are all based on the collection of publicly available results. Specifically, these include:
[0041] Here is an example of introducing an example:
[0042] 1.I will give you some additional information. The first part theadditional information is
[0043] 2. Some PLM candidates:
[0044] 3.[1]bert-base-uncased
[0045] 4.[2]bert-base-cased
[0046] 5.[3]roberta-base
[0047] 6.[4]dmis-lab / biobert-base-cased-v1.1
[0048] 7.[5]google / electra-base-discriminator
[0049] 8.[6]princeton-nlp / unsup-simcse-bert-base-uncased
[0050] 9.[7]princeton-nlp / sup-simcse-bert-base-uncased
[0051] 10.[8]openai-gpt
[0052] 11.[9]facebook / bart-base
[0053] 12.
[10] allenai / scibert_scivocab_cased
[0054] 13.
[11] allenai / scibert_scivocab_uncased
[0055] 14.
[12] distilbert-base-cased
[0056] 15.
[13] distilbert-base-uncased
[0057] 16.
[14] nghuyong / ernie-2.0-base-en
[0058] 17.
[15] distilroberta-base
[0059] 18.
[16] distilgpt2
[0060] 19.
[17] distilbert-base-multilingual-cased
[0061] 20.
[18] albert-base-v2
[0062] 21.
[19] microsoft / BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext
[0063] 22.
[20] michiyasunaga / BioLinkBERT-base
[0064] 23.The second part of the additional information is exampled rankingresults of models
[0065] 24.mentioned above on a given dataset:
[0066] 25.Dataset:SciFact.Models:[8,10,13,15,17].Ranking:[10,13,8,17,15]
[0067] 26.Dataset:SciFact.Models:[2,3,8,19,20].Ranking:[20,19,2,3,8]
[0068] 27.Dataset:SciFact.Models:[1,9,11,12,15].Ranking:[11,9,1,12,15]
[0069] 28.Dataset:SciFact.Models:[1,5,8,17,19].Ranking:[19,1,8,17,5]
[0070] 29.Dataset:SciFact.Models:[3,4,5,14,19].Ranking:[19,4,14,3,5]
[0071] 30.Dataset:MuTual.Models:[1,3,10,14,19].Ranking:[19,14,10,1,3]
[0072] 31.Dataset:MuTual.Models:[3,11,12,16,18].Ranking:[11,3,12,18,16]
[0073] 32.Dataset:MuTual.Models:[1,2,11,19,20].Ranking:[20,19,11,1,2]
[0074] 33.Dataset:MuTual.Models:[1,4,5,11,15].Ranking:[11,4,1,15,5]
[0075] 34.Dataset:MuTual.Models:[10,12,14,18,20].Ranking:[20,14,10,12,18]
[0076] For ease of representation, lines 3-22 are numbered for the example candidate models. Lines 25-34 are provided. Here are two examples. Line 25 is the first example, where... For SciFact, a list of example candidate models The values are {8, 10, 13, 15, 17}, where The specific name of each example candidate model corresponds to the number in rows 3-22; the performance ranking of these 5 example candidate models on the SciFact dataset is [10, 13, 8, 17, 15].
[0077] In one embodiment, step S3 above—applying the thought chain to the prompts to provide step-by-step guidance for the large language model to complete the model retrieval task—involves the large language model outputting candidate pre-trained models based on the prompts. Top performance Model retrieval results The final model retrieval results specifically include:
[0078] The large language model normalizes the reasoning process with thought chain cues as follows: The model retrieval task can be completed through the following steps:
[0079] Step S31: Collect information about Information;
[0080] Step S32: Obtain each Features, structure, target tasks, applicable fields, and parameter settings;
[0081] Step S33: Analyze the provided example information;
[0082] Step S34: Generate A performance ranking list of candidate pre-trained models, with the top... A set of candidate pre-trained models ranked by performance As the final model retrieval result.
[0083] Here is an example of a thought process chain:
[0084] 1.You need to solve this problem in 4 steps:
[0085] 2.Step 1: Search information of the given Dataset.
[0086] 3.Step 2: Search characteristics, structure, target tasks, applieddomain and config settings for every given model.
[0087] 4.Step 3: Consider the additional information I gave you above.
[0088] 5.Step 4: Combine all the knowledge you got through step 1-3 andoutput the
[0089] 6. Ranking list.
[0090] 7.Now please output a brief ranking list of 50 models instead of explanation.
[0091] Among them, lines 2-6 correspond to steps S31-S34 of the thought chain.
[0092] In one embodiment, step S4 above: using two The dual encoder is constructed in query set and document collection The resulting sentence embeddings are encoded as follows: and and in and Based on the sample features Specifically, it includes:
[0093] For a given target dataset and a candidate model ,in, Indicates the sample size. , and Representing query, document, and respectively and Does it match?
[0094] Step S41: A dual encoder consisting of two candidate pre-trained models generates a query set. and document collection The sentence embeddings are represented as follows: and ;in , , The label represents the dimension of sentence embedding. ;
[0095] Step S42: Use formula (1) to... and Integration to obtain sample features ,
[0096] (1)
[0097] in, Representing Hadamaji, .
[0098] In one embodiment, step S5 above: utilizing a large language model in The above query set and document collection Encode the sentences to obtain their embeddings. and After introducing the whitening mechanism, the whitened sentence embeddings are obtained. and The soft label is obtained by calculating the normalized cosine similarity between the two. Specifically, it includes:
[0099] Step S51: Utilize a large language model in The above query set and document collection Encoding is performed, and the resulting sentence embeddings are as follows: and ,in, , , The sentence embedding dimension represents the encoding of a large language model; Representing a large language model;
[0100] Pre-trained models often generate anisotropic embedding spaces, which limits the expressiveness of sentence embeddings in downstream tasks. In this case, cosine similarity may become inaccurate due to anisotropy. Since whitening has been shown to effectively mitigate the adverse effects of anisotropy, this invention introduces the following whitening mechanism before calculating the transferability score.
[0101] Step S52: Define the whitening mechanism:
[0102] All sentence embeddings encoded by large language models Sentence embedding for initial mean estimation Covariance Matrix This can be expressed as formula (2). (3):
[0103] (2)
[0104] (3)
[0105] in, yes A dimensional vector of all 1s. It is the identity matrix. >0 is a very small positive number, preventing It becomes a singular matrix.
[0106] This allows us to obtain the whitened embedding. As shown in formula (4):
[0107] (4)
[0108] Step S53: ... , Mapped to via whitening mechanism , For queries and documents Calculate using formula (5) and Normalized cosine similarity between them;
[0109] (5)
[0110] in, yes Candidate documents, It is a decimal to avoid division by zero;
[0111] Using formula (5) produces overly smooth, soft labels, so an edge parameter is introduced. To increase the difference between positive and negative samples, as shown in formula (6):
[0112] (6)
[0113] in, yes and Soft tags between.
[0114] In one embodiment, step S6 above: Obtaining the features of the whitened sample The parameters used are Modeling with marginal likelihood and Compatibility between models; calculate the maximum marginal likelihood score of the soft label as the final transferability score for each model in the model ranking stage, and select the model with the highest score as the final model selection result, specifically including:
[0115] Step S61: Obtain the whitened sample features using formula (7) :
[0116] (7)
[0117] Step S62: To calculate the maximum marginal likelihood of the soft label, use the parameter Linear models are used to model text features. and tags To ensure compatibility between these methods, traditional approaches typically use logistic or linear regression to obtain optimal parameters. And assess likelihood However, this method is prone to overfitting; to address this issue, marginal likelihood is used. An assessment was conducted; among which, and This can be represented using a graphical model, with the parameters being... and Therefore, the marginal likelihood is as shown in formula (8):
[0118] (8)
[0119] when When it is a positive definite matrix, Therefore, the marginal likelihood can be simplified to a function of logarithms. , The function is shown in formula (9):
[0120] (9)
[0121] in, , ;
[0122] Step S63: Parameters and Maximize The decision was made to evaluate the process through alternating treatments. , And in fixed , Maximize , This process can be represented by formula (10). (14):
[0123] (10)
[0124] (11)
[0125] (12)
[0126] (13)
[0127] (14)
[0128] in, yes The singular values;
[0129] exist and After convergence, the log-maximum likelihood is obtained. ;
[0130] Step S64; Calculate the normalized log-maximum likelihood to obtain the maximum marginal likelihood score for the soft label: This score is used as the final transferability score of the model in the model ranking stage. and The initial values of all models are set to 1; finally, the candidate pre-trained model with the highest maximum marginal likelihood score of the soft label is selected as the final model selection result.
[0131] The framework in this invention is named TELLMe (Teaching and Exploiting Large Language Models). First, model retrieval is performed using large model hints to narrow down the candidate model pool. Next, a maximum marginal likelihood estimation from soft labels (Ease) is proposed to estimate the transferability of the retrieved models. The maximum marginal likelihood from soft labels generated by embedding sentences into the large model is used to rank the models, thereby identifying the optimal model.
[0132] Validation experiments of the TELLMe framework were first conducted on the biomedical question-answering datasets ReQA BioASQ 8b and ReQAB BioASQ 9b. Furthermore, the experimental results were also validated on the general datasets NQ and SQuAD, which are widely used for retrieval-based question-answering tasks. For model retrieval, [the following methods were employed]. Recall rate ( , The evaluation metric was used as the benchmark. For model ranking, the experimental metrics from previous retrieval-based question answering work were used as a baseline; specifically, the Mean Reciprocal Rank was employed. ), Accuracy ( , )and Recall rate ( , ) Measure the performance of the pre-trained language model. Then, use and Kendall's correlation coefficient between them The coefficient is used to evaluate EaSe.
[0133] Table 1. Experimental results of model retrieval on ReQA BioASQ 8b
[0134]
[0135] Table 2. Experimental results of model retrieval on ReQA BioASQ 9b
[0136]
[0137] Table 3 Experimental results of model retrieval on NQ
[0138]
[0139] Table 4 Experimental results of model retrieval on SQuAD
[0140]
[0141] The experimental results for model retrieval are detailed in Tables 1–4. The experiments first used Bing Chat to converse with the large model. Bing Chat offers three modes: Creative, Balanced, and Precise. Given that Precise mode provides more accurate, reliable, and concise answers, this mode was chosen to input various types of prompts. The baseline only includes the task description; CoT introduces thought chains on top of the baseline; ICL includes the task description and examples of model retrieval on two other datasets; ICL+CoT includes the task description, thought chains, and examples with the same settings as ICL. ICL+CoT is the method applied during the model retrieval phase. In Tables 1–4, when Bing Chat is selected as the large language model, the baseline, CoT, ICL, and ICL+CoT settings are compared, and the best experimental results are indicated in bold. When GPT-4 is selected as the large language model, ICL+CoT for GPT-4 and ICL+CoT for Bing Chat are compared, and the best experimental results are indicated by underline. Experimental results on Bing chat show that, compared to other types of prompts, the baseline consistently produces the lowest R@5, R@10, and R@15 across various datasets, validating the effectiveness of both ICL and CoT. Overall, the model retrieval method ICL+CoT based on the TELLMe framework of this invention achieves best performance on all datasets, indicating that combining ICL and CoT is more advantageous than using either method alone. However, it is noteworthy that in ReQA BioASQ 9b, the R@15 and R@20 metrics of ICL+CoT are lower than those of ICL and CoT individually. This difference may stem from the limited inference capabilities of large models for long text inputs. This experiment further compared the performance of more large models, including GPT-3.5 and GPT-4. As shown in Tables 1-4, GPT-4 exhibits superior performance and achieves the best experimental results due to its ability to handle longer prompts and more complex tasks. It is worth noting that, unlike GPT-3.5, Bing Chat can access the web to collect and provide real-time information, thus offering superior performance compared to GPT-3.5.
[0142] Table 5. Experimental results of model ranking on ReQA BioASQ 8b
[0143]
[0144] Table 6. Experimental results of model ranking on ReQA BioASQ 9b
[0145]
[0146] Table 7 Experimental results of model ranking on NQ
[0147]
[0148] Table 8 Experimental results of model ranking on SQuAD
[0149]
[0150] As can be seen from Table 5-8, on all four datasets, except for ReQA BioASQ 8b... The model ranking method EaSe based on the TELLMe framework of this invention achieves higher Kendall correlation coefficient scores than other methods on all four datasets. Specifically, for and The values of EaSe on ReQA BioASQ 8b were 0.0084 and 0.0084 higher than the baseline model LogME, respectively; for , and The values of EaSe on ReQA BioASQ 9b are 0.0379, 0.0379 and 0.0337 higher than LogME, respectively; on NQ, it exceeds LogME by 0.0884, 0.0821 and 0.0463, respectively; and on SQuAD, it exceeds LogME by 0.0758, 0.0758 and 0.0758, respectively.
Claims
1. A method for selecting a retrieval-based question-answering model based on large language model prompts and embeddings, characterized in that, Includes the following steps: Step S1: Prompts for building a large language model, including: the application scenario of the model selection task and the selected target dataset. and A set of candidate pre-trained models ; Step S2: For the aforementioned prompt, introduce Examples provide prior knowledge for large language models to enhance their performance in model selection tasks; wherein, the examples include: example datasets. List of example candidate models and example candidate models in The performance ranking list on the platform, among which The number of candidate models contained in a given example; Step S3: Apply the thought chain to the prompts to provide step-by-step guidance for the large language model to complete the model retrieval task. Based on the prompts, the large language model outputs the candidate pre-trained model. Top performance Model retrieval results As the final model retrieval result; Step S4: Using two The dual encoder is constructed in query set and document collection The resulting sentence embeddings are encoded as follows: and and in and Based on the sample features ; Step S5: Utilize the large language model in The above query set and document collection Encode the sentences to obtain their embeddings. and After introducing the whitening mechanism, the whitened sentence embeddings are obtained. and The soft label is obtained by calculating the normalized cosine similarity between the two. Specifically, it includes: Step S51: Utilize the large language model in The above query set and document collection Encoding is performed, and the resulting sentence embeddings are as follows: and ,in, , , The sentence embedding dimension represents the encoding of a large language model; Representing a large language model; Step S52: Define the whitening mechanism: All sentence embeddings encoded by large language models Sentence embedding for initial mean estimation Covariance Matrix This can be expressed as formulas (2) to (3): (2) (3) in, Indicates the sample size; yes A dimensional vector of all 1s. It is the identity matrix. >0 is a very small positive number, preventing It becomes a singular matrix; This allows us to obtain the whitened embedding. As shown in formula (4): (4) Step S53: ... , Mapped to via whitening mechanism , For queries and documents Calculate using formula (5) and Normalized cosine similarity between them; (5) in, yes Candidate documents, It is a decimal to avoid division by zero; Using formula (5) produces overly smooth, soft labels, so an edge parameter is introduced. To increase the difference between positive and negative samples, as shown in formula (6): (6) in, yes and Soft tags between; Step S6: Obtain the features of the whitened sample The parameters used are Modeling with marginal likelihood and The compatibility between them is assessed; the maximum marginal likelihood score of the soft label is calculated as the final transferability score of each model in the model ranking stage, and the model with the highest score is selected as the final model selection result.
2. The retrieval-based question-answering model selection method based on large language model prompts and embeddings according to claim 1, characterized in that, Step S3: Applying the thought chain to the prompts provides step-by-step guidance for the large language model to complete the model retrieval task. Based on the prompts, the large language model outputs the candidate pre-trained model. Top performance Model retrieval results The final model retrieval results specifically include: The large language model normalizes the reasoning process with thought chain cues as follows: The model retrieval task can be completed through the following steps: Step S31: Collect information about Information; Step S32: Obtain each Features, structure, target tasks, applicable fields, and parameter settings; Step S33: Analyze the provided example information; Step S34: Generate A performance ranking list of candidate pre-trained models, with the top... A set of candidate pre-trained models ranked by performance As the final model retrieval result.
3. The retrieval-based question-answering model selection method based on large language model prompts and embeddings according to claim 2, characterized in that, Step S4: Using two The dual encoder is constructed in query set and document collection The resulting sentence embeddings are encoded as follows: and and in and Based on the sample features Specifically, it includes: For a given target dataset and a candidate model ,in, Indicates the sample size. , and Representing query, document, and respectively and Does it match? Step S41: A dual encoder consisting of two candidate pre-trained models generates a query set. and document collection The sentence embeddings are represented as follows: and ;in , , The label represents the dimension of sentence embedding. ; Step S42: Use formula (1) to... and Integration to obtain sample features : (1) in, Representing Hadamaji, .
4. The retrieval-based question-answering model selection method based on large language model prompts and embeddings according to claim 3, characterized in that, Step S6: Obtain the features of the whitened sample. The parameters used are Modeling with marginal likelihood and Compatibility between them; The maximum marginal likelihood score of the soft label is calculated as the final transferability score for each model in the model ranking stage. The model with the highest score is selected as the final model selection result. Specifically, this includes: Step S61: Obtain the whitened sample features using formula (7) : (7) Step S62: To calculate the maximum marginal likelihood of the soft label, use the parameters Linear models are used to model text features. and tags To ensure compatibility between these methods, traditional approaches typically use logistic or linear regression to obtain optimal parameters. And assess the likelihood. However, this method is prone to overfitting; to address this issue, marginal likelihood is used. An assessment was conducted; among which, and This can be represented using a graphical model, with the parameters being... and Therefore, the marginal likelihood is as shown in formula (8): (8) when When it is a positive definite matrix, Therefore, the marginal likelihood can be simplified to a function of logarithms. , The function is shown in formula (9): (9) in, , ; Step S63: Parameters and Maximize The decision was made to evaluate the process through alternating treatments. , And in fixed , Maximize , This process can be expressed as formulas (10) to (14): (10) (11) (12) (13) (14) in, yes The singular values; exist and After convergence, the log-maximum likelihood is obtained. ; Step S64; Calculate the normalized log-maximum likelihood to obtain the maximum marginal likelihood score for the soft label: And this is used as the final transferability score of the model in the model ranking stage, where and The initial values of all models are set to 1; finally, the candidate pre-trained model with the highest maximum marginal likelihood score of the soft label is selected as the final model selection result.