Search enhancement generation methods and apparatus, computer program products, electronic equipment

Abstract

This disclosure relates to a retrieval enhancement generation method and apparatus, computer program product, and electronic device, belonging to the field of artificial intelligence technology. The method includes: determining an input set of a target model and dividing the input set into a training set and a validation set; determining a candidate set and a result set, and determining parameter information for each element in the candidate set based on the training set to determine the elements to be added to the result set; validating each element in the result set based on the validation set to determine the target number of retrieval results; and inputting the retrieval results corresponding to the target number of retrieval results into the target model to generate output results. This disclosure can accurately determine the target number of retrieval results used for retrieval enhancement, improving the retrieval enhancement generation effect.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically, to a retrieval enhancement generation method, a retrieval enhancement generation device, a computer program product, and an electronic device. Background Technology

[0002] Retrieval-enhanced generation refers to the use of additional knowledge information to effectively address the problem of inaccurate results caused by untimely updates to training data when large models generate results.

[0003] In related technologies, the top few most relevant search results are selected as additional knowledge information and input into a large model to generate results.

[0004] Selecting too few search results may not cover the necessary knowledge information, while selecting too many may result in irrelevant or erroneous information being input into the model. Therefore, an inaccurate number of search results leads to inaccurate selection of knowledge information, which in turn affects the effectiveness of search enhancement generation. Summary of the Invention

[0005] The purpose of this disclosure is to provide a retrieval enhancement generation method, retrieval enhancement generation device, computer program product, and electronic device, thereby overcoming, at least to some extent, the problem of poor retrieval enhancement generation results caused by the limitations and defects of related technologies.

[0006] According to one aspect of this disclosure, a retrieval enhancement generation method is provided, comprising: determining an input set of a target model and dividing the input set into a training set and a validation set; determining a candidate set and a result set, and determining parameter information of each element in the candidate set based on the training set to determine the elements to be added to the result set; performing validation processing on each element in the result set based on the validation set to determine the target number of retrieval results; and inputting the retrieval results corresponding to the target number of retrieval results into the target model to generate output results.

[0007] In one exemplary embodiment of this disclosure, determining the parameter information of each element in the candidate set based on the training set to determine the element to be added to the result set includes: determining the test probability of each element in the candidate set based on the training set, and adding the element to the result set based on the test probability.

[0008] In an exemplary embodiment of this disclosure, determining the test probability of each element in the candidate set based on the training set includes: configuring a target error risk rate and determining an error risk rate; determining a first test probability and a second test probability of each element based on the target error risk rate, the number of elements in the training set, and the error risk rate; and determining the test probability of each element based on the minimum of the first test probability and the second test probability.

[0009] In one exemplary embodiment of this disclosure, the element is added to the result set based on the test probability, and the element is added to the result set when the test probability satisfies a probability condition; wherein the probability condition is determined based on a reference probability that makes the actual error risk rate less than the target error risk rate and the number of elements in the candidate set.

[0010] In one exemplary embodiment of this disclosure, determining the error risk rate includes: for each input in the training set, selecting relevant results using the model's RAG function as RAG results; for each input, determining the correlation between the RAG results and the input, and determining the number of relevant results in the training set; and determining the error risk rate based on the number of relevant results, the number of elements in the training set, and the number of selected RAG results.

[0011] In one exemplary embodiment of this disclosure, each element in the result set is validated based on the validation set to determine the target number of search results, including: determining the error risk rate of each element in the result set based on the validation set; and selecting the element with the smallest error risk rate in the result set as the target number of search results.

[0012] In one exemplary embodiment of this disclosure, the method further includes: using a K-fold cross-validation partitioning method to partition the training set and the validation set according to multiple different partitioning methods, and obtaining multiple search result quantity values ​​based on the partitioned multiple training sets and multiple validation sets; and taking the search result quantity value that appears most frequently among the multiple search result quantity values ​​as the target search result quantity.

[0013] According to one aspect of this disclosure, a retrieval enhancement generation apparatus is provided, comprising: a set partitioning module, configured to determine an input set of a target model and partition the input set into a training set and a validation set; an element addition module, configured to determine a candidate set and a result set, and determine parameter information of each element in the candidate set based on the training set to determine the elements to be added to the result set; a quantity determination module, configured to perform validation processing on each element in the result set based on the validation set to determine the target number of retrieval results; and a model processing module, configured to input the retrieval results corresponding to the target number of retrieval results into the target model and generate output results.

[0014] According to one aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the retrieval enhancement generation method described in any of the preceding claims.

[0015] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the retrieval enhancement generation method described in any one of the preceding claims by executing the executable instructions.

[0016] In the technical solution provided in this disclosure, on the one hand, by setting a candidate set and a result set, and determining the elements to be added to the result set based on the parameter information of each element in the candidate set according to the training set, and then evaluating the elements added to the result set to determine the final number of target search results, the method avoids having too many or too few search results, thereby improving the accuracy and matching of the determined number of target search results and enhancing the reliability of the determined number of target search results. On the other hand, since a suitable number of target search results can be selected, the effect of search enhancement can be improved.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0019] Figure 1 The flowchart of a retrieval enhancement generation method is illustrated schematically in an embodiment of the present disclosure.

[0020] Figure 2This illustration schematically shows a diagram of determining the test probability of each element in the candidate set in an embodiment of the present disclosure.

[0021] Figure 3 This diagram illustrates the determination of the error risk rate for each element.

[0022] Figure 4 The schematic diagram illustrates the generation of retrieval enhancements in an embodiment of this disclosure.

[0023] Figure 5 A schematic block diagram of a retrieval enhancement generation apparatus is shown in an embodiment of this disclosure.

[0024] Figure 6 A schematic block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0026] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0027] In some embodiments, Retrieval-Augmented Generation (RAG) effectively addresses the inaccuracy of large models' generated results due to untimely updates to training data by using additional knowledge information. Related techniques typically employ RAG to select the top 3-5 most relevant search results as additional knowledge information input into the large model for result generation. The number of search results selected directly impacts the accuracy of the large model's results; too few results may not cover the necessary knowledge information, while too many results may lead to irrelevant or erroneous information being input into the model. Therefore, a reliable method is needed to determine the number of search results selected.

[0028] To address the aforementioned technical issues, this disclosure provides a retrieval-augmented generation method. This method determines the number of target retrieval results to be selected during the retrieval-augmented generation process, and then performs retrieval-augmented generation based on the retrieval results corresponding to the selected number of target retrieval results. Retrieval-augmented generation (RAG) is a technique that combines information retrieval with language generation. Its purpose is to enhance the quality and accuracy of text generated by a language model by utilizing external knowledge sources (such as document databases, knowledge graphs, etc.). The retrieval-augmented generation method can be applied to scenarios such as content creation assistance, question-answering systems, dialogue systems, automatic report generation, and intelligent customer service.

[0029] Figure 1 The flowchart of the retrieval enhancement method is illustrated in the figure. (See reference) Figure 1 As shown, the retrieval enhancement generation method mainly includes the following steps:

[0030] In step S110, the input set of the target model is determined, and the input set is divided into a training set and a validation set;

[0031] In step S120, a candidate set and a result set are determined, and the parameter information of each element in the candidate set is determined based on the training set, so as to determine the elements to be added to the result set;

[0032] In step S130, each element in the result set is validated based on the validation set to determine the number of target search results;

[0033] In step S140, the search results corresponding to the target number of search results are input into the target model to generate output results.

[0034] The technical solution provided in this disclosure, on the one hand, by setting a candidate set and a result set, and determining the elements to be added to the result set based on the parameter information of each element in the candidate set according to the training set, and then evaluating the elements added to the result set to determine the final number of target retrieval results, avoids too many or too few retrieval results, and improves the accuracy and matching of the determined number of target retrieval results, thus improving the reliability of determining the number of target retrieval results. On the other hand, since the parameter information of the elements in the candidate set can be determined through the training set, and the elements to be added to the result set can be determined based on the parameter information, and an appropriate number of target retrieval results can be selected based on the error risk rate of each element in the result set, irrelevant or erroneous information is avoided from being input into the model, and the situation of missing the required information is also avoided, thereby improving the effect of retrieval enhancement.

[0035] Next, refer to Figure 1 The following provides a detailed description of the retrieval enhancement generation method in the embodiments of this disclosure.

[0036] In step S110, the input set of the target model is determined, and the input set is divided into a training set and a validation set.

[0037] In this embodiment, the target model can be a large model, such as a large language model. Retrieval enhancement generation refers to selecting multiple retrieval results as additional knowledge information and inputting this additional knowledge information into the target model to generate results. Using additional knowledge information can effectively solve the problem of inaccurate results caused by untimely updates to training data when the target model generates results. However, the number of retrieval results selected can affect the accuracy of the target model's results.

[0038] RAG (Related Aspects of Entities) technology enables a target model to generate answers that rely not only on its internal knowledge but also on external information sources, such as databases or knowledge bases, to provide more accurate and richer responses. RAG technology can include multiple steps: retrieval, enhancement, and generation. During the retrieval process, relevant contextual information can be retrieved from external knowledge sources based on the user's query. For example, an embedding model can be used to convert the user query into a vector, and a similarity search can be performed in a vector database to find the k most relevant results, which are then used as additional input to the target model.

[0039] In some embodiments, to accurately determine the number of search results to be selected, the target model can be assumed to be M, and the input to the target model can be represented as x. It can be assumed that multiple search results are retrieved using RAG technology as additional input to the target model. The number of multiple search results can be k. Based on this, the specific value of k can be determined to determine the number of search results used for enhancement.

[0040] Based on this, we can first determine the input set X of the target model, and then divide the input set into a training set X. train , verification set X valid With test set X test The input set can be a sufficiently random distribution. The partitioning ratio can be determined according to actual needs to divide the input set into training, validation, and test sets.

[0041] It should be noted that the input set can be partitioned once to obtain the training set, validation set, and test set; in addition, the training set and validation set can be partitioned multiple times according to different partitioning methods based on the K-fold cross-validation method.

[0042] In step S120, a candidate set and a result set are determined, and parameter information of each element in the candidate set is determined based on the training set, and the element to be added to the result set is determined based on the parameter information.

[0043] In this embodiment of the disclosure, after dividing the input set into a training set, a validation set, and a test set, the target number of search results can be determined based on the training set and validation set obtained from a single partition. Alternatively, the training set and validation set can be partitioned multiple times using a K-fold cross-validation method to obtain multiple search result counts; furthermore, the target number of search results can be determined based on these multiple counts.

[0044] First, the process of determining the number of target retrieval results based on the training set and validation set after a single partition is explained.

[0045] In some embodiments, a candidate set C is set. K Candidate set C K Each element in the set is a positive integer, for example, a positive integer between 3 and 10. Candidate set C K Each element in the list represents the number of search results to be tested. Meanwhile, a result set R can be defined. K The result set R K Used to store the number of search results that meet the test requirements.

[0046] When determining the target number of retrieval results based on the training and validation sets, parameter information for each element in the candidate set can be determined based on the training set. This parameter information can then be used to evaluate whether an element can be added to the result set, thus determining which elements to add from the candidate set. For example, the parameter information can be the test probability of each element in the candidate set, where the test probability refers to the probability used to evaluate whether each element in the candidate set can be added to the result set. Further, the addition of an element to the result set can be determined based on its test probability. It should be noted that for each element in the candidate set, parameter information can be determined based on the training set, thereby determining whether the element can be added to the result set.

[0047] Figure 2 The diagram illustrates the determination of the test probability for each element in the candidate set. (See reference...) Figure 2 As shown, the main steps include:

[0048] In step S210, the target error risk rate is configured and the error risk rate is determined;

[0049] In step S220, based on the target error risk rate and the error risk rate, the first test probability and the second test probability of each element are determined;

[0050] In step S230, the test probability of each element in the candidate set is determined based on the minimum of the first test probability and the second test probability.

[0051] In this embodiment of the disclosure, a target error risk rate α can be set. The target error risk rate can be set according to actual needs, for example, the target error risk rate can be set to 0.1. The target error risk rate is used to generate the first test probability of each element in the candidate set, and can also be used to generate the second test probability of each element in the candidate set.

[0052] For each element in the candidate set, the error risk rate can be determined. The error risk rate refers to the total error rate of the entire training set X. train Choose k for each input i Error risk rate for each RAG result. Figure 3 The flowchart illustrating the determination of the error risk rate for each element is shown in the image. (See reference...) Figure 3 As shown, the main steps include:

[0053] In step S310, for each input in the training set, a result related to the input is selected as the RAG result;

[0054] In step S320, for each input, the correlation between the RAG result and the input is determined, and the number of relevant results in the training set is determined.

[0055] In step S330, the error risk rate is determined based on the number of relevant results, the number of elements in the training set, and the number of selected RAG results.

[0056] In this embodiment of the disclosure, for each element in the candidate set, the following operations can be performed: For each input in the training set, multiple results related to each input in the training set are selected using the RAG function of the target model, and these results are used as the RAG results for each input. Exemplarily, different RAG algorithms select results related to each input in the training set in different ways. In some embodiments, relevant contextual information can be retrieved from external knowledge sources based on the user's query. Exemplarily, an embedding model can be used to convert the user query into a vector, and a similarity search can be performed in a vector database to find the most relevant information as multiple results related to the input, thereby using these multiple results related to the input as the RAG results.

[0057] For each input, the relevance between the RAG result and the input is determined to ascertain the number of relevant results in the training set. For example, a correlation coefficient can be calculated between the RAG result and the input. RAG results whose correlation coefficients satisfy a correlation condition are considered relevant results corresponding to the input. Further, the number of relevant results corresponding to each input can be determined, and the total number of relevant results in the training set can be determined based on the sum of the number of relevant results corresponding to each input. The correlation coefficient can be a parameter representing the degree of relevance, and it can be of any type. A correlation condition could be, for example, a correlation coefficient greater than a correlation threshold, which can be determined based on actual needs. Of course, other conditions can also be used, depending on the specific requirements.

[0058] Furthermore, the error risk rate can be determined based on the number of relevant results, the number of elements in the training set, and the number of selected RAG results. The number of selected RAG results refers to the number of results related to each input in the training set selected through the RAG function of the target model. For example, the error risk rate can be determined by calculating the product of the number of elements in the training set and the number of selected RAG results, and by calculating the ratio between the number of relevant results and the product. The formula for calculating the error risk rate is shown in formula (1):

[0059] FR(k i )=1-K rel / (k i *|X train |) Formula (1)

[0060] Among them, K rel The number of relevant results in the entire training set, |X train | represents the training set X train The number of elements in the middle, k i The number of RAG results selected.

[0061] After determining the error risk rate, for each element in the candidate set, two hypothesis test probabilities can be calculated. These two probability tests can include a first test probability and a second test probability. For example, the first and second test probabilities for each element in the candidate set can be determined based on the set target error risk rate and the calculated error risk rate.

[0062] For example, the first test probability is used to represent the probability calculated based on the 0-1 distribution. The first test probability can be determined based on the error risk rate, the set target error risk rate, and the number of elements in the training set. For example, the first test probability can be obtained by calculating the exponent based on the product of the divergence between the error risk rate and the target error risk rate and the number of elements in the training set. The formula for calculating the first test probability is as shown in formula (2):

[0063]

[0064] in,

[0065] H(a,b)=a*log(a / b)+(1-a)*log(1-a) / (1-b)) is used to represent the Kullback-Leibler divergence between two Bernoulli distributions, and here it is used to represent the divergence between the error risk rate and the target error risk rate.

[0066] The second test probability is used to represent the probability that the product of the number of occurrences of a random event and the number of elements in the training set is less than the error risk rate. The binomial distribution is a distribution that satisfies the condition that the probability of a random event occurring is equal to the target error risk rate, and the number of repetitions is equal to the number of elements in the training set. The second test probability can be determined based on the error risk rate, the set target error risk rate, and the number of elements in the training set. The formula for calculating the second test probability is shown in formula (3):

[0067]

[0068] Where P(X) represents the probability of finding X, and Bin(X) represents the probability of finding X. train |,α) refers to a random event with probability α and repetition count |X. train The binomial distribution of |X| is given by the expression for the entire P, which is essentially finding the binomial distribution when the number of random events is less than |X|.train |*FR(k i The probability of ( ).

[0069] Furthermore, the minimum of the first and second test probabilities can be determined as the test probability of each element in the candidate set. For example, the test probability can be determined according to formula (4):

[0070]

[0071] Based on this, after determining the test probability of each element in the candidate set according to the minimum of the first test probability and the second test probability, it can be determined whether the test probability of each element in the candidate set satisfies the probability condition. The probability condition can be determined according to the pre-set reference probability and the number of elements in the candidate set. The reference probability δ can be the probability that makes the actual error risk rate less than the target error risk rate. For example, the value of the reference probability can be 0.95, or it can be determined according to actual needs. No specific limitation is made here. The probability condition can be determined according to the ratio between the difference between the fixed parameter and the reference probability and the number of elements in the candidate set. The probability condition can be expressed as formula (5):

[0072] P ki ≤(1-δ) / |C K | Formula (5)

[0073] If the test probability of any element in the training set satisfies the probability condition, then the elements in the training set that satisfy the probability condition can be added to the result set R. K Repeat the above steps to determine whether the test probability of each element in the training set meets the probability condition. If it does, add multiple elements in the training set whose test probabilities meet the probability condition to the result set to obtain the number of search results that meet the test requirements.

[0074] In step S130, each element in the result set is validated based on the validation set to determine the number of target retrieval results.

[0075] In this embodiment of the disclosure, after determining the elements that can be added to the result set from the candidate set based on the training set, each element in the result set can be verified based on the validation set to determine the number of target retrieval results, thereby realizing the determination of the number of target retrieval results based on each element in the result set.

[0076] For example, for all elements in the result set, the error risk rate of each element in the result set can be calculated based on the validation set, and the final number of target search results can be determined based on the error risk rate of each element in the result set.

[0077] Based on this, the error risk rate can be calculated using the same method as in the training set. For example, for each element in the validation set, for each input in the validation set, multiple relevant results are selected using the model's RAG function to serve as the RAG results of the validation set; for each input, the correlation between the RAG results of the validation set and the input is determined to identify the number of relevant results in the validation set; based on the number of relevant results, the number of elements in the validation set, and the number of selected RAG results, the error risk rate of each element in the result set is determined.

[0078] Furthermore, the element with the lowest error risk rate in the result set can be selected as the final number of target search results, which is the value of k.

[0079] After determining the target number of search results, the results can be validated on a test set. For example, the test set X can be... test For each input, use the candidate set C K Each element k in i The error risk rate for each element is calculated. The method for calculating the error risk rate for each element is the same as the method for calculating the error risk rate for each element in the candidate set based on the training set. Furthermore, the selected number of target retrieval results k can be validated based on the error risk rate of each element. t Is the error risk rate small enough?

[0080] In this embodiment, when dividing the input set into a training set and a validation set, a K-fold cross-validation partitioning method can be used to partition the training set and validation set multiple times according to various different partitioning methods, resulting in multiple training sets and multiple validation sets. Further, based on the multiple partitioned training sets and multiple validation sets, the parameter information of each element in the candidate set is determined to determine the elements to be added to the result set; each element in the result set is validated based on the validation set to determine the number of search results. Based on this, multiple search result counts can be obtained. On this basis, the search result count value that appears most frequently among the multiple search result counts can be selected as the final target number of search results. It should be noted that when calculating each search result count value, it is still possible to follow... Figure 1 The calculation is performed in the manner described in the text.

[0081] For example, the kfold partitioning method can be used to partition the training and validation sets N times according to various different partitioning methods, resulting in N values ​​for the number of search results, which can be represented as k. t1 k t2 ...k tN Furthermore, the search result count that appears most frequently among the N search result counts can be selected as the final target search result count k. tOf course, other methods can also be used to determine the final target number of search results from multiple search result counts; no specific limitations are made here.

[0082] In step S140, the search results corresponding to the target number of search results are input into the target model to generate output results.

[0083] In this embodiment of the disclosure, after obtaining the target number of search results, the search results corresponding to the target number of search results can be input into the target model to generate output results. The search results can be additional knowledge information. For example, the search results corresponding to the target number of search results can be combined with the user's original query to form a new enhanced prompt, which serves as a prompt word to guide the target model in generating an answer. The enhanced prompt word is then input into the target model to generate the final answer, thus obtaining the output result. For example, if the target number of search results is 8, 8 search results can be input into the target model to obtain the output result.

[0084] refer to Figure 4 As shown, if the target number of search results is 4, the search results for that target number of search results can be input into the target model for search enhancement. For example, inputting search result 1, search result 2, search result 3, and search result 4 into the target model will yield the output results.

[0085] The technical solution provided in this disclosure selects the target number of search results that minimizes the overall error risk through statistical testing, avoiding the limitations caused by selecting too many or too few search results. By performing hypothesis testing on a set of candidate search result numbers, the target number of search results with the lowest error risk rate is selected based on the training and validation sets, given the target model structure and parameters, and the current RAG knowledge base distribution, effectively improving the final performance of RAG.

[0086] This disclosure also provides a retrieval enhancement generation apparatus. (See reference) Figure 5 As shown, the retrieval enhancement generation device 500 mainly includes the following modules:

[0087] The set partitioning module 501 is used to determine the input set of the target model and partition the input set into a training set and a validation set;

[0088] The element addition module 502 is used to determine the candidate set and the result set, and to determine the parameter information of each element in the candidate set based on the training set, so as to determine the element to be added to the result set;

[0089] The quantity determination module 503 is used to perform verification processing on each element in the result set based on the verification set to determine the number of target search results;

[0090] The model processing module 504 is used to input the search results corresponding to the target number of search results into the target model and generate output results.

[0091] In one exemplary embodiment of this disclosure, determining the parameter information of each element in the candidate set based on the training set to determine the element to be added to the result set includes: determining the test probability of each element in the candidate set based on the training set, and adding the element to the result set based on the test probability.

[0092] In an exemplary embodiment of this disclosure, determining the test probability of each element in the candidate set based on the training set includes: configuring a target error risk rate and determining an error risk rate; determining a first test probability and a second test probability of each element based on the target error risk rate, the number of elements in the training set, and the error risk rate; and determining the test probability of each element based on the minimum of the first test probability and the second test probability.

[0093] In one exemplary embodiment of this disclosure, the element is added to the result set based on the test probability, and the element is added to the result set when the test probability satisfies a probability condition; wherein the probability condition is determined based on a reference probability that makes the actual error risk rate less than the target error risk rate and the number of elements in the candidate set.

[0094] In one exemplary embodiment of this disclosure, determining the error risk rate includes: for each input in the training set, selecting relevant results using the model's RAG function as RAG results; for each input, determining the correlation between the RAG results and the input, and determining the number of relevant results in the training set; and determining the error risk rate based on the number of relevant results, the number of elements in the training set, and the number of selected RAG results.

[0095] In one exemplary embodiment of this disclosure, each element in the result set is validated based on the validation set to determine the target number of search results, including: determining the error risk rate of each element in the result set based on the validation set; and selecting the element with the smallest error risk rate in the result set as the target number of search results.

[0096] In one exemplary embodiment of this disclosure, the apparatus further includes: using a K-fold cross-validation partitioning method to partition the training set and the validation set according to a variety of different partitioning methods, and obtaining a variety of search result quantity values ​​based on the partitioned training set and the multiple validation set; and taking the search result quantity value that appears most frequently among the multiple search result quantity values ​​as the target search result quantity.

[0097] It should be noted that the specific details of each module in the above-mentioned search enhancement generation device have been described in detail in the corresponding search enhancement generation method, so they will not be repeated here.

[0098] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0099] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0100] In an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0101] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0102] The following reference Figure 6 To describe an electronic device 600 according to such an embodiment of the present disclosure. Figure 6 The electronic device 600 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0103] like Figure 6 As shown, the electronic device 600 is manifested in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including storage unit 620 and processing unit 610), and a display unit 640.

[0104] The storage unit stores program code that can be executed by the processing unit 610, causing the processing unit 610 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 610 can perform actions such as... Figure 1 The steps are shown in the figure.

[0105] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0106] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0107] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0108] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. As shown, network adapter 660 communicates with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0109] It should be noted that some embodiments of this disclosure also provide a computer program product, which includes a computer program that implements the above-described method when executed by a processor.

[0110] In one embodiment, the computer program product can be a tangible product containing a computer program, such as a computer-readable storage medium storing the computer program. The readable storage medium can be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory, hard disk drive (HDD), solid-state drive (SSD), etc. For example, the computer program product can be implemented as a non-volatile storage medium storing the computer program, such as read-only memory, NAND flash memory, etc. In one embodiment, the computer program product can be an intangible product containing a computer program. For example, the computer program product can be implemented as a virtual digital product, such as an executable file, installation package, or other digital file storing the computer program.

[0111] Computer program code can be written in one or more programming languages. Examples of programming languages ​​include C, Java, and C++. Program code can execute entirely on the user's computing device, partially on the user's computing device, or as a standalone software package. It can also execute partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, such as a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via an internet connection provided by a mobile network operator).

[0112] Computer programs can be carried or transmitted via signals such as electrical, magnetic, optical, electromagnetic, and infrared rays. Electronic devices can convert signals carrying computer programs into digital signals, thereby running the computer programs. When a computer program runs on an electronic device, its code is used to cause the electronic device to execute (more specifically, to be executed by the processor of the electronic device) the method steps of various exemplary embodiments of this disclosure.

[0113] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0114] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0115] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0116] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0117] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for retrieval augmentation generation, the method comprising: The method comprises the following steps: determining an input set of a target model, and dividing the input set into a training set and a verification set; determining a candidate set and a result set, configuring a target error risk rate and determining an error risk rate; determining a first test probability and a second test probability of each element based on the target error risk rate, the number of elements in the training set and the error risk rate; determining a test probability of each element in the candidate set according to the minimum value of the first test probability and the second test probability, and adding the element to the result set based on the test probability; verifying each element in the result set based on the verification set to determine a target number of search results; inputting the search results corresponding to the target number of search results into the target model to generate an output result; wherein the first test probability and the second test probability of each element are determined as follows: determining an index according to the product of the divergence between the error risk rate and the target error risk rate and the number of elements in the training set to perform exponential calculation to obtain the first test probability; determining the second test probability according to the error risk rate, the target error risk rate and the number of elements in the training set.

2. The search enhancement generation method of claim 1, wherein, The adding of the element to the result set based on the test probability comprises: adding the element to the result set when the test probability meets a probability condition; wherein the probability condition is determined based on a reference probability that makes the actual error risk rate less than the target error risk rate and the number of elements in the candidate set.

3. The search enhancement generation method of claim 1, wherein, The determination of the error risk rate comprises: selecting relevant results as RAG results through the RAG function of the model for each input in the training set; determining the number of relevant results in the training set by judging the relevance of the RAG results and the input for each input respectively; determining the error risk rate based on the number of relevant results, the number of elements in the training set and the number of selected RAG results.

4. The search enhancement generation method of claim 1, wherein, The verification of each element in the result set based on the verification set to determine the target number of search results comprises: determining the error risk rate of each element in the result set based on the verification set; taking the element with the minimum error risk rate in the result set as the target number of search results.

5. The search enhancement generation method of claim 1, wherein, The method further comprises: dividing the training set and the verification set according to multiple different division manners by using the K-fold cross-validation division method, and obtaining multiple number of search results based on the divided multiple training sets and multiple verification sets; taking the number of search results that appears most frequently in the multiple number of search results as the target number of search results.

6. A retrieval augmentation generation apparatus characterized by comprising: The method comprises the following steps: a set division module for determining an input set of a target model, and dividing the input set into a training set and a verification set; an element adding module for setting a candidate set and a result set, configuring a target error risk rate and determining an error risk rate; determining a first test probability and a second test probability of each element based on the target error risk rate, the number of elements in the training set and the error risk rate; determining a test probability of each element in the candidate set according to a minimum value of the first test probability and a second test probability, and adding the element to the result set based on the test probability; a quantity determination module, configured to determine a target search result quantity by performing verification processing on each element in the result set based on the verification set; a model processing module, configured to input a search result corresponding to the target search result quantity into a target model to generate an output result; wherein the first test probability and the second test probability of each element are determined as follows: determining an index according to a product of a divergence between the error risk rate and the target error risk rate and a number of elements in the training set, and performing exponential calculation to obtain the first test probability; determining the second test probability according to the error risk rate, the target error risk rate, and the number of elements in the training set.

7. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the search enhancement generation method of any one of claims 1-5.

8. An electronic device, comprising: comprise: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the search enhancement generation method of any one of claims 1-5 by executing the executable instructions.

Images