Retrieval enhancement generation method and device, computer program product and electronic equipment

Abstract

The embodiment of the invention relates to a retrieval enhancement generation method and device, a computer program product and electronic equipment, and relates to the technical field of artificial intelligence. The method comprises the steps that an input set of a target model is determined, and the input set is divided into a training set and a verification set; determining a candidate set and a result set, and determining parameter information of each element in the candidate set based on the training set so as to determine the elements added into the result set; performing verification processing on each element in the result set based on the verification set, and determining the number of target retrieval results; inputting retrieval results corresponding to the number of the target retrieval results into a target model to generate an output result. According to the method, the number of the target retrieval results for retrieval enhancement can be accurately determined, and the retrieval enhancement generation effect is improved.

Description

Background Art

[0002] Retrieval-enhanced generation refers to the use of additional knowledge information to effectively solve the problem of inaccurate results caused by untimely updates of training data when large models generate results.

[0003] In related technologies, the first few most relevant search results are selected as additional knowledge information and input into the large model for result generation.

[0004] If too few search results are selected, the required knowledge information may not be covered; if too many search results are selected, irrelevant or erroneous information may be input into the model. Therefore, the inaccurate number of search results leads to inaccurate selected knowledge information, which in turn affects the effect of search enhancement generation. Summary of the invention

[0005] The purpose of the present disclosure is to provide a retrieval enhancement generation method, a retrieval enhancement generation device, a computer program product and an electronic device, thereby overcoming the problem of poor retrieval enhancement generation effect caused by the limitations and defects of related technologies at least to a certain extent.

[0006] According to one aspect of the present disclosure, a retrieval enhancement generation method is provided, including: determining an input set of a target model, and dividing the input set into a training set and a verification set; determining a candidate set and a result set, and determining parameter information of each element in the candidate set based on the training set to determine the elements to be added to the result set; performing verification processing on each element in the result set based on the verification set to determine the target number of retrieval results; inputting the retrieval results corresponding to the target number of retrieval results into the target model to generate output results.

[0007] In an exemplary embodiment of the present disclosure, determining the parameter information of each element in the candidate set based on the training set to determine the elements to be added to the result set includes: determining the test probability of each element in the candidate set based on the training set, and adding the element to the result set based on the test probability.

[0008] In an exemplary embodiment of the present disclosure, determining the test probability of each element in the candidate set based on the training set includes: configuring a target error risk rate and determining the error risk rate; determining a first test probability and a second test probability of each element based on the target error risk rate, the number of elements in the training set and the error risk rate; and determining the test probability of each element according to the minimum value of the first test probability and the second test probability.

[0009] In an exemplary embodiment of the present disclosure, the element is added to the result set based on the test probability, and the element is added to the result set when the test probability satisfies a probability condition; wherein the probability condition is determined based on a reference probability that makes the actual error risk rate less than the target error risk rate and the number of elements in the candidate set.

[0010] In an exemplary embodiment of the present disclosure, determining the error risk rate includes: for each input in the training set, selecting relevant results through the RAG function of the model as RAG results; for each input, separately judging the correlation between the RAG result and the input, and determining the number of relevant results in the training set; determining the error risk rate based on the number of relevant results, the number of elements in the training set, and the number of selected RAG results.

[0011] In an exemplary embodiment of the present disclosure, each element in the result set is verified based on the verification set to determine the target number of retrieval results, including: determining the error risk rate of each element in the result set based on the verification set; and taking the element with the smallest error risk rate in the result set as the target number of retrieval results.

[0012] In an exemplary embodiment of the present disclosure, the method also includes: using a K-fold cross-validation partitioning method to divide the training set and the validation set according to a plurality of different partitioning methods, and obtaining a plurality of retrieval result quantity values ​​based on the divided plurality of training sets and the plurality of validation sets; and taking the retrieval result quantity value that appears the most times among the plurality of retrieval result quantity values ​​as the target retrieval result quantity.

[0013] According to one aspect of the present disclosure, a retrieval enhancement generation device is provided, including: a set partitioning module, used to determine an input set of a target model, and divide the input set into a training set and a verification set; an element adding module, used to determine a candidate set and a result set, and determine parameter information of each element in the candidate set based on the training set to determine the elements to be added to the result set; a quantity determination module, used to perform verification processing on each element in the result set based on the verification set to determine the number of target retrieval results; and a model processing module, used to input the retrieval results corresponding to the target number of retrieval results into the target model to generate output results.

[0014] According to one aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the above-mentioned retrieval enhancement generation methods.

[0015] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned retrieval enhancement generation methods by executing the executable instructions.

[0016] In the technical solution provided in the embodiment of the present disclosure, on the one hand, by setting a candidate set and a result set, and determining the elements to be added to the result set based on the parameter information of each element in the candidate set in the training set, and then evaluating and processing the elements to be added to the result set, the number of target search results finally selected is determined, thereby avoiding too many or too few search results, improving the accuracy and matching of the determined number of target search results, and improving the reliability of determining the number of target search results. On the other hand, since the appropriate number of target search results can be selected, the effect of search enhancement can be improved.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0019] Figure 1 A flowchart of a retrieval enhancement generation method in an embodiment of the present disclosure is schematically shown.

[0020] Figure 2 The schematic diagram schematically shows a method of determining the test probability of each element in the candidate set in an embodiment of the present disclosure.

[0021] Figure 3 A schematic diagram schematically illustrates the determination of the error risk rate for each element.

[0022] Figure 4 The schematic diagram of retrieval enhancement generation in an embodiment of the present disclosure is schematically shown.

[0023] Figure 5 The block diagram of the retrieval enhancement generation device in the embodiment of the present disclosure is schematically shown.

[0024] Figure 6 The block diagram of the electronic device according to the embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0026] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0027] In some embodiments, Retrieval-Augmented Generation can effectively solve the problem of inaccurate results caused by untimely updating of training data when the large model generates results by using additional knowledge information. In the related art, the top 3-5 most relevant retrieval results are generally selected through RAG technology as additional knowledge information and input into the large model for result generation. The number of selected retrieval results has a direct impact on the accuracy of the results of the large model. If too few retrieval results are selected, the required knowledge information may not be covered. If too many retrieval results are selected, it is likely that irrelevant or erroneous information will be input into the model. Therefore, a reliable method is needed to determine the number of retrieval results to be selected.

[0028] In order to solve the above technical problems, in an embodiment of the present disclosure, a retrieval-augmented generation method is provided, which is used to determine the number of target retrieval results that need to be selected in the retrieval-augmented generation process, and then perform retrieval-augmented generation according to the retrieval results corresponding to the selected number of target retrieval results. Retrieval-Augmented Generation (RAG) is a technology that combines information retrieval with language generation. Its purpose is to enhance the quality and accuracy of text generated by the language model by utilizing external knowledge sources (such as document databases, knowledge graphs, etc.). The retrieval-augmented generation method can be applied to scenarios such as content creation assistance, question-and-answer systems, dialogue systems, automatic report generation, and intelligent customer service.

[0029] Figure 1 A flowchart of the retrieval enhancement method is schematically shown in FIG. Figure 1 As shown in , the retrieval enhancement generation method mainly includes the following steps:

[0030] In step S110, an input set of a target model is determined, and the input set is divided into a training set and a validation set;

[0031] In step S120, a candidate set and a result set are determined, and parameter information of each element in the candidate set is determined based on the training set to determine the elements to be added to the result set;

[0032] In step S130, each element in the result set is verified based on the verification set to determine the number of target search results;

[0033] In step S140, the search results corresponding to the target number of search results are input into the target model to generate output results.

[0034] The technical solution provided in the embodiments of the present disclosure, on the one hand, by setting a candidate set and a result set, and determining the elements to be added to the result set based on the parameter information of each element in the candidate set according to the training set, and then evaluating and processing the elements to be added to the result set, and determining the number of target search results to be finally selected, avoids too many or too few search results, can improve the accuracy and matching of the determined number of target search results, and improves the reliability of determining the number of target search results. On the other hand, because it is possible to determine the parameter information of the elements in the candidate set through the training set, and then determine the elements to be added to the result set based on the parameter information, and select the appropriate number of target search results based on the error risk rate of each element in the result set, it is avoided that irrelevant or erroneous information is input into the model, and it is also avoided that the required information is omitted, and the effect of search enhancement can be improved.

[0035] Next, refer to Figure 1The retrieval enhancement generation method in the embodiment of the present disclosure is specifically described as shown in FIG.

[0036] In step S110, an input set of a target model is determined, and the input set is divided into a training set and a validation set.

[0037] In the disclosed embodiments, the target model may be a large model, such as a large language model. Retrieval enhancement generation refers to selecting multiple retrieval results as additional knowledge information, and inputting the additional knowledge information into the target model for result generation. Among them, the use of additional knowledge information can effectively solve the problem of inaccurate results caused by untimely updating of training data when the target model generates results. However, the number of selected retrieval results may affect the accuracy of the results of the target model.

[0038] RAG technology is used to enable the target model to rely not only on its internal knowledge when generating answers, but also to use external information sources, such as databases or knowledge bases, to provide more accurate and richer answers. RAG technology may include multiple steps of retrieval, enhancement, and generation. During the retrieval process, relevant contextual information can be retrieved from external knowledge sources based on the user's query. Exemplarily, an embedding model can be used to convert the user query into a vector, and a similarity search can be performed in the vector database to find the most relevant information as k retrieval results, thereby using the k retrieval results as additional input to the target model.

[0039] In some embodiments, in order to accurately determine the number of selected search results, it can be assumed that the target model is M, the input of the target model is represented as x, and it is assumed that multiple search results are retrieved through the RAG technology as additional inputs of the target model. The number of multiple search results can be k. Based on this, the specific value of k can be determined to determine the number of search results for search enhancement.

[0040] Based on this, we can first determine the input set X of the target model, and then divide the input set into the training set X train , validation set X valid With the test set X test The input set may be an input set with sufficiently random distribution. The partition ratio may be determined according to actual needs, so as to divide the input set into a training set, a validation set, and a test set according to the partition ratio.

[0041] It should be noted that the input set can be divided once to obtain a training set, a validation set, and a test set; in addition, the training set and the validation set can be divided multiple times in different ways according to the K-fold cross-validation partitioning method.

[0042] In step S120, a candidate set and a result set are determined, and parameter information of each element in the candidate set is determined based on the training set, and elements to be added to the result set are determined according to the parameter information.

[0043] In the disclosed embodiment, after the input set is divided into a training set, a validation set, and a test set, the target number of search results can be determined based on the training set and the validation set divided once. In addition, the training set and the validation set can be divided multiple times according to different division methods according to the K-fold cross-validation division method to obtain multiple search result quantity values; further, the target number of search results can be determined based on the multiple search result quantity values ​​obtained by the multiple divisions.

[0044] First, the process of determining the number of target retrieval results based on a once-divided training set and a validation set is described.

[0045] In some embodiments, the candidate set C is set K , candidate set C K Each element in is a positive integer, for example, a positive integer between 3 and 10. K Each element in is used to represent the number of search results to be checked. At the same time, a result set R can be set K , the result set R K Used to store the number of search results that meet the inspection requirements.

[0046] When determining the target number of retrieval results based on the training set and the validation set, the parameter information of each element in the candidate set can be determined based on the training set, so as to evaluate whether the element can be added to the result set based on the parameter information of each element, so as to determine the elements to be added to the result set from the candidate set. Exemplarily, the parameter information can be the test probability of each element in the candidate set, and the test probability refers to the probability used to evaluate whether each element in the candidate element set can be added to the result set. Further, it can be determined whether to add the element to the result set based on the test probability of each element. It should be noted that for each element in the candidate set, the parameter information of the element in the candidate set can be determined based on the training set, so as to determine whether the element can be added to the result set based on the parameter information of the element.

[0047] Figure 2 A schematic diagram of determining the test probability of each element in the candidate set is shown schematically in Figure 2 As shown in , it mainly includes the following steps:

[0048] In step S210, a target error risk rate is configured and the error risk rate is determined;

[0049] In step S220, based on the target error risk rate and the error risk rate, a first inspection probability and a second inspection probability of each element are determined;

[0050] In step S230, the test probability of each element in the candidate set is determined according to the minimum value of the first test probability and the second test probability.

[0051] In the embodiment of the present disclosure, a target error risk rate α can be set, and the target error risk rate can be set according to actual needs, for example, the target error risk rate can be set to 0.1. The target error risk rate is used to generate a first test probability for each element in the candidate set, and can also be used to generate a second test probability for each element in the candidate set.

[0052] For each element in the candidate set, the error risk rate can be determined. The error risk rate refers to the error rate of the entire training set X. train Each input of i The risk of error when the RAG result is Figure 3 A flowchart for determining the error risk rate for each element is schematically shown in FIG. Figure 3 As shown in , it mainly includes the following steps:

[0053] In step S310, for each input in the training set, a result related to the input is selected as a RAG result;

[0054] In step S320, for each input, the correlation between the RAG result and the input is determined, and the number of relevant results in the training set is determined;

[0055] In step S330, the error risk rate is determined based on the number of the relevant results, the number of elements in the training set, and the number of selected RAG results.

[0056] In an embodiment of the present disclosure, the following operation can be performed for each element in the candidate set: for each input in the training set, multiple results related to each input in the training set are selected through the RAG function of the target model as the RAG result of each input. Exemplarily, different RAG algorithms select results related to each input in the training set in different specific ways. In some embodiments, relevant contextual information can be retrieved from an external knowledge source based on the user's query. Exemplarily, an embedding model can be used to convert the user query into a vector, and a similarity search can be performed in the vector database to find the most relevant information as multiple results related to the input, thereby using the multiple results related to the input as RAG results.

[0057] For each input, the correlation between the RAG result and the input is determined respectively, and the number of relevant results in the training set is determined. For example, the correlation coefficient between the RAG result and the input can be calculated, and the RAG result whose correlation coefficient meets the correlation condition is used as the relevant result corresponding to the input. The number of relevant results corresponding to each input can be further determined, and then the number of relevant results in the training set is determined according to the sum of the number of relevant results corresponding to each input. The correlation coefficient can be a parameter used to represent the degree of correlation, and the correlation coefficient can be any type of parameter. The correlation condition can, for example, be that the correlation coefficient is greater than the correlation threshold, and the correlation threshold can be determined according to actual needs. Of course, the correlation condition can also be other conditions, which are determined specifically according to actual needs.

[0058] Furthermore, the error risk rate can be determined based on the number of relevant results, the number of elements in the training set, and the number of selected RAG results. The number of selected RAG results refers to the number of results related to each input in the training set selected by the RAG function of the target model. For example, the product of the number of elements in the training set and the number of selected RAG results can be calculated, and the ratio between the number of relevant results and the product can be calculated to determine the error risk rate. The calculation formula of the error risk rate can be shown in formula (1):

[0059] FR(k i )=1-K rel / (k i *|X train |) Formula (1)

[0060] Among them, K rel is the number of relevant results in the entire training set, |X train | represents the training set X train The number of elements in k i is the number of selected RAG results.

[0061] After determining the error risk rate, two hypothesis test probabilities may be calculated for each element in the candidate set. The two hypothesis test probabilities may include a first test probability and a second test probability. Exemplarily, the first test probability and the second test probability of each element in the candidate set may be determined based on the set target error risk rate and the calculated error risk rate.

[0062] Exemplarily, the first test probability is used to represent the probability calculated based on the 0-1 distribution, and the first test probability can be determined according to the error risk rate, the set target error risk rate, and the number of elements in the training set. Exemplarily, an exponent can be determined according to the product of the divergence between the error risk rate and the target error risk rate and the number of elements in the training set to perform exponential calculation to obtain the first test probability. The calculation formula of the first test probability can be shown as formula (2):

[0063]

[0064] in,

[0065] H(a,b)=a*log(a / b)+(1-a)*log(1-a) / (1-b)), which is used to represent the Kullback-Leibler divergence between two Bernoulli distributions, and is used here to represent the divergence between the error risk rate and the target error risk rate.

[0066] The second test probability is used to represent the probability of binomial distribution when the number of occurrences of a random event is less than the product of the number of elements in the training set and the error risk rate. Binomial distribution refers to a distribution that satisfies the probability of a random event occurrence as the target error risk rate and the number of repetitions as the number of elements in the training set. The second test probability can be determined based on the error risk rate, the set target error risk rate, and the number of elements in the training set. The calculation formula for the second test probability can be shown in formula (3):

[0067]

[0068] Among them, P(X) represents the probability of finding X, Bin(X train |,α) refers to the probability of a random event occurring is α, and the number of repetitions is |X train | binomial distribution, the expression of the entire P is to find the binomial distribution when the number of random events is less than |X train |*FR(k i ) is the probability when .

[0069] Furthermore, the minimum value of the first test probability and the second test probability can be determined as the test probability of each element in the candidate set. For example, the test probability can be determined according to formula (4):

[0070]

[0071] On this basis, after determining the test probability of each element in the candidate set according to the minimum value of the first test probability and the second test probability, it can be determined whether the test probability of each element in the candidate set meets the probability condition. The probability condition can be determined based on the pre-set reference probability and the number of elements in the candidate set, where the reference probability δ can be the probability that the actual error risk rate is less than the target error risk rate. The value of the reference probability can be, for example, 0.95, or can be determined according to actual needs, which is not specifically limited here. The probability condition can be determined based on the difference between the fixed parameter and the reference probability and the ratio between the number of elements in the candidate set. The probability condition can be expressed as formula (5):

[0072] P ki ≤(1-δ) / |C K | Formula (5)

[0073] If the test probability of any element in the training set meets the probability condition, the element in the training set that meets the probability condition can be added to the result set R K Repeat the above steps to determine whether the test probability of each element in the training set meets the probability condition. If so, multiple elements in the training set whose test probability meets the probability condition can be added to the result set to obtain the number of retrieval results that meet the test requirements.

[0074] In step S130, each element in the result set is verified based on the verification set to determine the target number of retrieval results.

[0075] In an embodiment of the present disclosure, after determining the elements in the candidate set that can be added to the result set based on the training set, each element in the result set can be verified based on the verification set to determine the target number of retrieval results, thereby achieving the determination of the target number of retrieval results based on each element in the result set.

[0076] Exemplarily, for all elements in the result set, the error risk rate of each element in the result set may be calculated based on the validation set, and the number of target retrieval results finally selected may be determined based on the error risk rate of each element in the result set.

[0077] Based on this, the error risk rate can be calculated in the same way as in the training set. Exemplarily, for each element in the validation set, for each input in the validation set, multiple relevant results are selected through the RAG function of the model as the RAG results of the validation set; for each input, the correlation between the RAG results of the validation set and the input is determined, and the number of relevant results in the validation set is determined; based on the number of relevant results, the number of elements in the validation set, and the number of selected RAG results, the error risk rate of each element in the result set is determined.

[0078] Furthermore, the element with the smallest error risk rate in the result set may be selected as the final selected target retrieval result quantity, that is, as the value of k.

[0079] After determining the target number of search results, the results can be verified on the test set. test For each input on K Each element k in i , calculate the error risk rate corresponding to each element. The method of calculating the error risk rate of each element is the same as the method of calculating the error risk rate of each element in the candidate set based on the training set. Furthermore, the selected target retrieval result number k can be verified based on the error risk rate of each element. t Is the error risk rate small enough?

[0080] In the embodiment of the present disclosure, when dividing the input set into a training set and a validation set, a K-fold cross-validation partitioning method can be used to divide the training set and the validation set multiple times according to a variety of different partitioning methods to obtain multiple training sets and multiple validation sets. Furthermore, based on the divided multiple training sets and multiple validation sets, the parameter information of each element in the candidate set is determined to determine the elements to be added to the result set; each element in the result set is verified based on the validation set to determine the number of retrieval results. Based on this, multiple retrieval result numbers can be obtained. On this basis, the retrieval result number that appears the most times among the multiple retrieval result numbers can be selected as the final target retrieval result number. It should be noted that when calculating the number of each retrieval result, it can still be calculated according to Figure 1 Calculated in the way.

[0081] For example, the kfold partitioning method is used to partition the training set and the validation set N times according to various partitioning methods, and N search result values ​​are obtained, which can be expressed as k t1 , k t2 ...k tN Furthermore, the retrieval result value with the largest number of occurrences among the N retrieval result values ​​can be selected as the final target retrieval result value k. t Of course, the final target number of search results may also be determined from multiple search result quantity values ​​in other ways, which are not specifically limited here.

[0082] In step S140, the search results corresponding to the target number of search results are input into the target model to generate output results.

[0083] In the disclosed embodiment, after obtaining the target number of search results, the search results corresponding to the target number of search results can be input into the target model to generate output results. The search results can be additional knowledge information. Exemplarily, the search results corresponding to the target number of search results retrieved can be combined with the user's original query to form a new enhanced prompt as a prompt word to guide the target model to generate an answer based on the prompt word. The enhanced prompt word is input into the target model to generate a final answer to obtain an output result. For example, if the target number of search results is 8, 8 search results can be input into the target model to obtain an output result.

[0084] refer to Figure 4 As shown in , if the target number of search results is 4, the search results of the target number of search results can be input into the target model for search enhancement. For example, search result 1, search result 2, search result 3, and search result 4 are input into the target model to obtain output results.

[0085] The technical solution provided in the embodiments of the present disclosure selects the target number of search results that can minimize the overall error risk by means of statistical testing, thereby avoiding the limitations caused by selecting too many or too few search results. By performing hypothesis testing on a set of candidate search result values, the target number of search results with the lowest error risk rate under the target model structure and parameters, and the current RAG knowledge base distribution is selected based on the training set and the validation set, thereby effectively improving the final effect of RAG.

[0086] The present disclosure also provides a search enhancement generation device. Figure 5 As shown, the search enhancement generation device 500 mainly includes the following modules:

[0087] A set partitioning module 501 is used to determine an input set of a target model and to partition the input set into a training set and a validation set;

[0088] An element adding module 502 is used to determine a candidate set and a result set, and determine parameter information of each element in the candidate set based on the training set to determine an element to be added to the result set;

[0089] A quantity determination module 503 is used to perform verification processing on each element in the result set based on the verification set to determine the number of target search results;

[0090] The model processing module 504 is used to input the search results corresponding to the target search result quantity into the target model to generate output results.

[0091] In an exemplary embodiment of the present disclosure, determining the parameter information of each element in the candidate set based on the training set to determine the elements to be added to the result set includes: determining the test probability of each element in the candidate set based on the training set, and adding the element to the result set based on the test probability.

[0092] In an exemplary embodiment of the present disclosure, determining the test probability of each element in the candidate set based on the training set includes: configuring a target error risk rate and determining the error risk rate; determining a first test probability and a second test probability of each element based on the target error risk rate, the number of elements in the training set and the error risk rate; and determining the test probability of each element according to the minimum value of the first test probability and the second test probability.

[0093] In an exemplary embodiment of the present disclosure, the element is added to the result set based on the test probability, and the element is added to the result set when the test probability satisfies a probability condition; wherein the probability condition is determined based on a reference probability that makes the actual error risk rate less than the target error risk rate and the number of elements in the candidate set.

[0094] In an exemplary embodiment of the present disclosure, determining the error risk rate includes: for each input in the training set, selecting relevant results through the RAG function of the model as RAG results; for each input, separately judging the correlation between the RAG result and the input, and determining the number of relevant results in the training set; determining the error risk rate based on the number of relevant results, the number of elements in the training set, and the number of selected RAG results.

[0095] In an exemplary embodiment of the present disclosure, each element in the result set is verified based on the verification set to determine the target number of retrieval results, including: determining the error risk rate of each element in the result set based on the verification set; and taking the element with the smallest error risk rate in the result set as the target number of retrieval results.

[0096] In an exemplary embodiment of the present disclosure, the device also includes: using a K-fold cross-validation partitioning method to divide the training set and the validation set according to a plurality of different partitioning methods, and obtaining a plurality of retrieval result quantity values ​​based on the divided plurality of training sets and the plurality of validation sets; and taking the retrieval result quantity value that appears the most times among the plurality of retrieval result quantity values ​​as the target retrieval result quantity.

[0097] It should be noted that the specific details of each module in the above-mentioned retrieval enhancement generation device have been described in detail in the corresponding retrieval enhancement generation method, so they will not be repeated here.

[0098] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0099] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0100] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0101] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0102] Refer to the following Figure 6 The electronic device 600 according to this embodiment of the present disclosure is described. Figure 6 The electronic device 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0103] like Figure 6 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: the at least one processing unit 610, the at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), and a display unit 640.

[0104] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0105] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0106] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0107] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0108] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 660. As shown, the network adapter 660 communicates with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0109] It should be noted that, in some embodiments of the present disclosure, a computer program product is also provided. The computer program product includes a computer program. When the computer program is executed by a processor, the above method is implemented.

[0110] In one embodiment, a computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing a computer program. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, and other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid-state drive (SSD), and the like. Exemplarily, a computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory (Nand Flash), and the like. In one embodiment, a computer program product may be an intangible product containing a computer program. Exemplarily, a computer program product may be implemented as a virtual digital product, such as a digital file such as an executable file, an installation package, etc. storing a computer program.

[0111] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0112] The computer program may be carried or transmitted via electrical, magnetic, optical, electromagnetic, infrared, or other signals. The electronic device may convert the signal carrying the computer program into a digital signal, thereby running the computer program. When the computer program is run on an electronic device, its code is used to enable the electronic device to execute (more specifically, the processor of the electronic device may execute) the method steps of various exemplary embodiments of the present disclosure.

[0113] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0114] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0115] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0116] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing what is disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be considered merely as exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.

[0117] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A search enhancement generation method, characterized in that: include: Determine an input set of a target model, and divide the input set into a training set and a validation set; Determine a candidate set and a result set, and determine parameter information of each element in the candidate set based on the training set to determine the elements to be added to the result set; Performing verification processing on each element in the result set based on the verification set to determine the number of target retrieval results; The search results corresponding to the target search result quantity are input into the target model to generate output results.

2. The search enhancement generation method according to claim 1, characterized in that: The determining, based on the training set, parameter information of each element in the candidate set to determine the elements to be added to the result set comprises: A test probability of each element in the candidate set is determined based on the training set, and the element is added to the result set based on the test probability.

3. The search enhancement generation method according to claim 2, characterized in that: The step of determining the test probability of each element in the candidate set based on the training set includes: Configure the target error risk rate and determine the error risk rate; Determine a first test probability and a second test probability for each element based on the target error risk rate, the number of elements in the training set, and the error risk rate; The test probability of each element is determined according to a minimum value of the first test probability and the second test probability.

4. The search enhancement generation method according to claim 2, characterized in that: adding the element to the result set based on the test probability, When the test probability satisfies the probability condition, adding the element to the result set; The probability condition is determined based on a reference probability that makes the actual error risk rate less than the target error risk rate and the number of elements in the candidate set.

5. The search enhancement generation method according to claim 3, characterized in that: Determining the error risk rate includes: For each input in the training set, the relevant result is selected through the RAG function of the model as the RAG result; For each input, determine the relevance of the RAG result to the input and determine the number of relevant results in the training set; The error risk rate is determined based on the number of correlation results, the number of elements in the training set, and the number of selected RAG results.

6. The search enhancement generation method according to claim 1, characterized in that: The step of performing verification processing on each element in the result set based on the verification set to determine the number of target search results includes: Determining the error risk rate of each element in the result set based on the validation set; The element with the smallest error risk rate in the result set is taken as the target number of retrieval results.

7. The search enhancement generation method according to claim 1, characterized in that: The method further comprises: The K-fold cross-validation partitioning method is used to partition the training set and the validation set according to a variety of different partitioning methods, and multiple retrieval result quantity values ​​are obtained based on the multiple divided training sets and multiple validation sets; The retrieval result quantity value that appears the most times among the multiple retrieval result quantity values ​​is used as the target retrieval result quantity.

8. A search enhancement generation device, characterized in that: include: A set partitioning module is used to determine an input set of a target model and divide the input set into a training set and a validation set; An element adding module, used to set a candidate set and a result set, and determine parameter information of each element in the candidate set based on the training set to determine the elements to be added to the result set; A quantity determination module, used to perform verification processing on each element in the result set based on the verification set to determine the number of target search results; The model processing module is used to input the search results corresponding to the target search result quantity into the target model to generate output results.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the retrieval enhancement generation method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the retrieval enhancement generation method according to any one of claims 1 to 7 by executing the executable instructions.

Images