Retrieval method, system and equipment based on semantic confusion and medium

Abstract

The invention discloses a semantic confusion-based retrieval method, system and device and a medium, and the method comprises the steps: before and after endowing retrieval information, sampling a given question based on a preset large language model, and obtaining a corresponding answer result and a generation probability; all the answer results expressing semantic equivalence are merged into the same semantic class set; calculating the sum of the generation probabilities of all the answer results in the semantic class set, and taking the sum as the semantic confusion of the large language model for the given question; measuring the contribution degree of the retrieval information by analyzing the gain between the determined semantic perplexity before and after endowing the retrieval information with the retrieval information; measuring the contribution degree of the retrieval information by analyzing the gain between the determined semantic perplexity before and after endowing the retrieval information with the retrieval information; and performing optimization training on the large language model according to an enhanced sample data set constructed by the contribution degree to improve the model output quality.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and information retrieval, and in particular to a retrieval method, system, device, and medium based on semantic perplexity. Background Technology

[0002] With the significant progress made by large-scale pre-trained language models in various natural language generation tasks, "Retrieval-Augmented Generation" (RAG) has gradually become a practical solution. Its core idea is to retrieve relevant information from external knowledge bases before or during the generation process and provide it to the model as context, in order to improve the performance of the generated content in terms of correctness, consistency, or completeness.

[0003] In RAG (Regenerative Analytics), the helpfulness of retrieval depends on the model's own knowledge and whether the retrieved content can truly help the model generate better results. Therefore, traditional retrieval evaluation metrics (such as NDCG, precision, and recall) are not entirely suitable for RAG scenarios. Existing technologies, when evaluating RAG systems, typically only focus on whether the final generated content meets the requirements, failing to independently measure the contribution of retrieval itself to the results. This makes it difficult to measure the utility of retrieval in generative tasks, resulting in relatively singular or one-sided model outputs.

[0004] Therefore, accurately measuring the actual contribution of retrieval and optimizing the output of the model have become technical problems that urgently need to be solved by those skilled in the art. Summary of the Invention

[0005] This invention provides a retrieval method, system, device, and medium based on semantic perplexity, to address how to characterize the knowledge gain of the retrieved information on the model through changes in semantic perplexity, providing a basis for optimizing the retrieval module and RAG workflow, and improving the accuracy of the language model's output results.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a retrieval method based on semantic perplexity, comprising:

[0007] Before assigning retrieval information, the given question is sampled based on a preset large language model to obtain the corresponding benchmark answer results and the first generation probability of the large language model generating each benchmark answer result.

[0008] After providing the retrieval information, the large language model is used to sample the given question and the target document set returned by the retrieval, and the corresponding enhanced answer results and the second generation probability of the large language model generating each of the enhanced answer results are obtained.

[0009] The baseline and enhanced response results are evaluated, and semantically equivalent responses are merged into the same semantic class set.

[0010] Based on the first generation probability and the second generation probability, the sum of the generation probabilities of all answer results in the corresponding semantic class set is calculated to determine the first semantic perplexity and the second semantic perplexity of the large language model for the given question before and after assigning retrieval information.

[0011] The contribution of the retrieved information is measured by analyzing the gain between the first semantic perplexity and the second semantic perplexity.

[0012] The large language model is optimized and trained based on the augmented sample dataset constructed from the contributions.

[0013] Furthermore, the step of determining the baseline answer and the enhanced answer, and merging semantically equivalent target answers into the same semantic class set, includes:

[0014] Based on a predetermined reference answer to the given question, the semantic equivalence of the reference answer and the enhanced answer is determined before and after the retrieval information is provided.

[0015] By performing cluster analysis on the judgment results, the answer results that express equivalent semantics to the reference answer are merged into the same semantic class set.

[0016] Furthermore, the determination of semantic equivalence between the baseline response and the enhanced response includes:

[0017] Based on the introduced dynamic threshold determination mechanism, the text entailment model is used to determine the semantic equivalence of the baseline response and the enhanced response.

[0018] Furthermore, the step of analyzing the gain between the first semantic perplexity and the second semantic perplexity includes the following formula:

[0019] ΔSePer=P M (a*|D)-P M (a*)

[0020] In the formula, ΔSePer is the information gain metric for retrieval; P M (a*|D) represents the likelihood probability of generating an answer after assigning search information; P M (a*) represents the likelihood probability of generating an answer before assigning search information; a* represents the reference answer.

[0021] Furthermore, the step of measuring the contribution of the retrieved information by analyzing the gain between the first semantic perplexity and the second semantic perplexity also includes:

[0022] In multi-hop reasoning scenarios, semantic perplexity is calculated independently for each step of the retrieval process;

[0023] The reasoning chain of the large language model is optimized based on the contribution of each step of the retrieval process determined by the calculation results.

[0024] Further, the step of optimizing and training the large language model based on the augmented sample dataset constructed from the contributions includes:

[0025] The enhanced sample dataset is constructed using high-contribution retrieval information determined based on the measurement results.

[0026] During training, the prompts and retrieval content generated by the large language model are optimized based on the augmented sample dataset.

[0027] Furthermore, the method also includes:

[0028] When the reference answer contains multiple correct answers, calculate the language perplexity gain corresponding to each of the correct answers;

[0029] We perform a weighted analysis on the calculation results to determine the contribution of multi-answer retrieval information.

[0030] Another embodiment of the present invention provides a retrieval system based on semantic perplexity, comprising:

[0031] The benchmark sampling module is used to sample a given question based on a preset large language model before assigning retrieval information, and to obtain the corresponding benchmark answer results and the first generation probability of the large language model generating each benchmark answer result.

[0032] An enhanced sampling module is used to sample the given question and the target document set returned by the retrieval using the large language model after the retrieval information is provided, so as to obtain each enhanced answer result and the second generation probability of each enhanced answer result generated by the large language model;

[0033] The semantic determination module is used to determine the baseline answer result and the enhanced answer result, and to merge the target answer results that express semantic equivalence into the same semantic class set;

[0034] The semantic perplexity analysis module is used to calculate the sum of the generation probabilities of all answer results in the corresponding semantic class set based on the first generation probability and the second generation probability, respectively, so as to determine the first semantic perplexity and the second semantic perplexity of the large language model for the given question before and after giving the retrieval information;

[0035] The retrieval contribution analysis module is used to measure the contribution of the retrieved information by analyzing the gain between the first semantic perplexity and the second semantic perplexity.

[0036] The model optimization module is used to optimize and train the large language model based on the augmented sample dataset constructed from the contributions.

[0037] Another embodiment of the present invention provides a computer device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the semantic perplexity-based retrieval method as described above.

[0038] In another embodiment of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, wherein when the device containing the computer-readable storage medium executes the computer program, the semantic perplexity-based retrieval method described above is implemented.

[0039] Compared with the prior art, the beneficial effects of the embodiments of the present invention are at least one of the following:

[0040] This invention defines "semantic perplexity" as an indicator to measure the contribution of retrieval. Specifically, it calculates the change in information entropy of the model before and after retrieval and uses this change as the semantic perplexity to measure the contribution of the retrieved information. This optimizes retrieval strategies, improves retrieval accuracy and relevance, and further enhances the accuracy and reliability of the large model's output. This invention only requires multiple samplings of existing large language models to estimate their subjective distribution of correct answers, without needing to call external large model APIs, making it convenient, fast, and cost-effective. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the retrieval method based on semantic perplexity in one embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram of the structure of a retrieval system based on semantic perplexity in one embodiment of the present invention;

[0043] Figure 3 A structural block diagram of a preferred embodiment of a computer device provided by the present invention;

[0044] Explanation of reference numerals in the attached figures:

[0045] M1, Baseline Sampling Module; M2, Enhanced Sampling Module; M3, Semantic Judgment Module; M4, Semantic Perplexity Analysis Module; M5, Retrieval Contribution Analysis Module; M6, Model Optimization Module. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0047] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0048] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. The terms "vertical," "horizontal," "left," "right," "upper," "lower," and similar expressions used herein are for illustrative purposes only and do not indicate or imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0049] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing specific embodiments only and is not intended to limit the invention. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0050] This invention aims to provide a method for quantifying the retrieval value of language models in generative scenarios that enhance retrieval capabilities. By characterizing the change in the language model's confidence in the correct (or optimal) answer before and after retrieval, it accurately measures the information gain brought to the model by external retrieval information, optimizes the retrieval module, and improves the output quality of large language models. Based on this, one embodiment of the invention provides a retrieval method based on semantic perplexity. By quantifying the contribution of retrieval in generative tasks (such as question answering, reasoning, and content creation), it optimizes the language model. For details, please refer to [link to specific documentation]. Figure 1 , Figure 1 The diagram shown illustrates a semantic perplexity-based retrieval method flowchart according to one embodiment of the present invention, including the following steps:

[0051] S1~S2, before and after assigning search information, the given question is sampled based on the preset large language model to obtain the corresponding answer results and generation probability.

[0052] Specifically, before assigning retrieval information, the given question is sampled multiple times based on a preset large language model to obtain the corresponding baseline answer results and the first generation probability of the large language model generating each baseline answer result.

[0053] After providing the retrieval information, the given question and the target document set returned by the retrieval are sampled multiple times using a large language model to obtain the corresponding enhanced answer results and the second generation probability of the large language model generating each enhanced answer result.

[0054] Specifically, the present invention provides a concrete example to further describe the above-mentioned solution:

[0055] First, in this example, the problem (or task) that needs to be input into the model is represented as q, and the set of target documents returned by the retrieval module is represented as: D = {d1, d2, ..., d...} n} represents the set of external documents retrieved, and the set of reference answers for q is represented as: A = {a*}.

[0056] (a) Before assigning retrieval information, perform baseline sampling of the question without using retrieval information. Specifically: allow the selected model M to sample the question q under a given temperature or random seed to generate multiple answers, and record the results r of each answer. i and its corresponding likelihood value l i The likelihood values ​​corresponding to each answer are transformed into a probability distribution form, such as by normalization.

[0057] (b) After providing the retrieval information, the question is augmented using the retrieval information. Specifically: the document set D and the question q are provided together to the large language model M to generate the answer. Similar to (a), sampling is repeated multiple times. The output answer and its corresponding likelihood probability are also recorded.

[0058] It is worth noting that, in order to balance computational cost and accuracy, the number of samplings N can be set between 5 and 10.

[0059] Understandably, the embodiments of the present invention aim to measure "information validity" from the perspective of changes in the information entropy of the language model before and after retrieval, and to solve the problem that traditional methods can only respond to the final answer or can only consider the surface matching of documents and answers.

[0060] S3-S4 First, in this embodiment, the baseline answer result and the enhanced answer result are determined, and the answer results that express semantic equivalence are merged into the same semantic class set.

[0061] Understandably, the embodiments of the present invention generate results based on multiple samplings of the question using a language model, and approximate the distribution of the model in the semantic space. Different expressions conveying the same semantic meaning are considered as the same answer event, and the probability of these "correct answer events" is statistically calculated to obtain the required semantic perplexity.

[0062] Specifically, this embodiment uses a pre-determined reference answer to a given question as a benchmark, and performs semantic equivalence determination on the benchmark answer and the enhanced answer before and after assigning retrieval information.

[0063] In some embodiments of the present invention, a dynamic threshold determination mechanism is introduced, which uses a textual entailment model (NLI) to determine the semantic equivalence of the response results. For example, a DeBERTa model fine-tuned for the textual entailment task can be used to determine whether the response is equivalent to the reference answer. Further, an adjustable confidence threshold can be introduced for the determination.

[0064] By performing cluster analysis on the judgment results, answers expressing semantically equivalent meaning to the reference answer are merged into the same semantic class set. In some embodiments of this invention, for the simplified version of SePer, the implied probability output by the NLI model and the probability of generating the model's answer can be directly weighted and summed to achieve an effect similar to fuzzy clustering. Alternatively, clustering algorithms (such as hierarchical clustering, K-means, etc.) can be used to analyze the similarity matrix and group semantically similar answers into the same cluster.

[0065] Furthermore, the sum of the generation probabilities of all answer results in the semantic class set is calculated as the semantic perplexity of the large language model for the given question. That is, based on the first generation probability, the sum of the generation probabilities of all answer results in the corresponding semantic class set is calculated as the first semantic perplexity of the large language model for the given question before assigning retrieval information. And based on the second generation probability, the sum of the generation probabilities of all answer results in the corresponding semantic class set is calculated as the second semantic perplexity of the large language model for the given question before assigning retrieval information.

[0066] For example, before assigning search information, if the answer result r is determined based on the judgment result... i If it is semantically equivalent to the reference answer a*, then it is considered a "correct answer", based on the corresponding likelihood probability e. i Calculate r i The sum of the probabilities of the semantic classes to which they belong is given by P. M (a*). Similarly, after assigning the retrieval information, performing the same operation yields P. M (a*|D).

[0067] That is, in the embodiments of the present invention, "P" M (a*)” and “P M "(a*|D)" represents the baseline values ​​of the model's confidence in the correct answer when the search information is not used and when the search information is included, respectively.

[0068] S5. By analyzing the gain between the first semantic perplexity and the second semantic perplexity, the contribution of the retrieved information is measured, and the contribution is used to characterize the utility of the retrieved information in the generation task.

[0069] Understandably, this embodiment uses "ΔSePer" (change in semantic perplexity) to characterize the knowledge gain of the retrieved information on the model, that is, to measure the difference in the model's confidence in the correct answer before and after retrieval, thereby reflecting the actual contribution of the retrieval more accurately and in a finer granular manner.

[0070] Specifically, the gain ΔSePer is defined as the difference between the two, expressed as follows:

[0071] ΔSePer=P M (a*|D)-P M (a*)

[0072] In the formula, ΔSePer is the information gain metric for retrieval; P M (a*|D) represents the likelihood probability of generating an answer after assigning search information; P M (a*) represents the likelihood probability of generating the answer before assigning search information.

[0073] It should be understood that if ΔSePer > 0, it indicates that the retrieval has brought a positive gain, meaning that the retrieval information can improve the accuracy of the language model's output; if it is less than or equal to 0, it indicates that the retrieval information is useless or has brought a negative gain, meaning that the retrieval has introduced interference. In other words, this gain can be used to quantify the utility of the retrieval information D.

[0074] It is worth noting that, in the face of multi-hop reasoning scenarios of language models, this embodiment can independently evaluate each retrieval step to characterize its fine-grained contribution to the final generation. In the reasoning chain, the contribution of each retrieval step can be evaluated using ΔSePer to better schedule and allocate retrieval resources, assisting in the optimization of the large model's reasoning chain. That is, in multi-hop reasoning scenarios, the semantic perplexity of each retrieval process is calculated independently, and the contribution of each retrieval process determined by the calculation results is used to optimize the reasoning chain of the large language model.

[0075] Furthermore, multiple-answer scenarios may occur, that is, when the reference answer contains multiple correct answers, such as the reference answer set A = {a1*, ..., a...}. k *}, at this point, each correct answer a can be calculated. j The language perplexity gain corresponding to *∈A is used to determine the contribution of retrieval information in multi-answer scenarios. The calculation results are then weighted or averaged to determine the contribution of retrieval information.

[0076] S6. Optimize and train the large language model based on the enhanced sample dataset constructed from the contribution.

[0077] This embodiment uses high-contribution retrieval information determined based on measurement results to construct an enhanced sample dataset.

[0078] During training, the prompts and retrieval content generated by the large language model are optimized based on the augmented sample dataset. This includes steps such as dynamic prompt design, retrieval guidance prompts, and retrieval strategy iteration. It should be understood that dynamic prompts (such as annotating key information) can guide the model to focus on high-value segments, while retrieval strategy iteration (such as query term optimization) can improve retrieval relevance.

[0079] This embodiment aims to use the calculated ΔSePer to filter out high-contribution retrieval information and add high-contribution (q,D,a*) pairs to the training set. During this process, a threshold can be set, such as selecting samples with ΔSePer>θ (e.g., 10%), to ensure that the quality of generated retrieval information is significantly improved. Alternatively, all samples can be sorted in descending order by ΔSePer, and the Top-K (e.g., Top 20%) can be selected as augmentation samples.

[0080] Understandably, ΔSePer directly quantifies the contribution of retrieved information to the generated results, and selecting high ΔSePer samples essentially means choosing cases where "retrieval information effectively assists the model in generating the correct answer." Therefore, by constructing augmented datasets through methods such as reorganizing retrieved information and adding adversarial noise, the model's ability to identify key information, its robustness, and its multimodal parsing capabilities can be effectively improved.

[0081] In other embodiments of this example, by comparing the ΔSePer of different retrieval generation systems, the retrieval parameters or strategies are selected to achieve a higher response gain for the RAG system at the same computational cost.

[0082] Alternatively, further analysis of which questions can be answered correctly without retrieval and which must rely on retrieval can help pinpoint potential knowledge gaps in large models, providing a reference for subsequent model training or fine-tuning.

[0083] In summary, this invention proposes a retrieval utility evaluation method based on "semantic perplexity," which measures retrieval utility by characterizing the difference in confidence of correct answers before and after retrieval in the semantic space. The defined ΔSePer can intuitively quantify the contribution of external documents to the language model output and can be widely used for evaluating various natural language generation tasks that require external knowledge enhancement. It provides new ideas for the design and optimization of retrieval modules and RAG systems. By optimizing the retrieval module and constructing augmented datasets through ΔSePer, the language model can be optimized, making the model's output answers more accurate and reliable.

[0084] One embodiment of the present invention provides a retrieval system based on semantic perplexity. For details, please refer to [link to documentation]. Figure 2 , Figure 2 The diagram shown illustrates the structure of a semantically perplexity-based retrieval system according to one embodiment of the present invention, including:

[0085] The benchmark sampling module M1 is used to sample the given question based on a preset large language model before assigning retrieval information, and obtain the corresponding benchmark answer results and the first generation probability of the large language model generating each benchmark answer result.

[0086] The enhanced sampling module M2 is used to sample the given question and the target document set returned by the retrieval using the large language model after the retrieval information is given, so as to obtain each enhanced answer result and the second generation probability of each enhanced answer result generated by the large language model;

[0087] The semantic determination module M3 is used to determine the baseline answer result and the enhanced answer result, and to merge the target answer results that express semantic equivalence into the same semantic class set;

[0088] The semantic perplexity analysis module M4 is used to calculate the sum of the generation probabilities of all answer results in the corresponding semantic class set based on the first generation probability and the second generation probability, respectively, so as to determine the first semantic perplexity and the second semantic perplexity of the large language model for the given question before and after giving the retrieval information;

[0089] The retrieval contribution analysis module M5 is used to measure the contribution of the retrieved information by analyzing the gain between the first semantic perplexity and the second semantic perplexity.

[0090] The model optimization module M6 is used to optimize and train the large language model based on the augmented sample dataset constructed from the contribution.

[0091] like Figure 3 As shown, this embodiment of the invention also provides a computer device. Figure 3 This is a structural block diagram of a preferred embodiment of a computer device provided by the present invention. The computer device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the semantic perplexity-based retrieval method as described above.

[0092] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device.

[0093] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor can be any conventional processor. The processor is the control center of the terminal device, connecting various parts of the terminal device through various interfaces and lines.

[0094] The memory mainly includes a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function, etc., while the data storage area can store related data, etc. Furthermore, the memory can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard drive, a SmartMedia Card (SMC), a Secure Digital (SD) card, and a Flash Card, or other volatile solid-state storage devices.

[0095] It should be noted that the aforementioned terminal devices may include, but are not limited to, processors and memory, as will be understood by those skilled in the art. Figure 3 The structural block diagram is merely an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown, or combine certain components, or use different components. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0096] Accordingly, embodiments of the present invention provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the steps in the method of the above embodiments, for example... Figure 1 Steps S1 to S6 as described above.

[0097] The technical features and effects of the semantic perplexity-based retrieval system proposed in this embodiment of the invention are the same as those of the semantic perplexity-based retrieval method proposed in this embodiment of the invention, and will not be repeated here.

[0098] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A search method based on semantic perplexity, characterized by, The method comprises the following steps: Before giving the retrieval information, a preset large language model is used to sample the given question to obtain corresponding baseline answer results and first generation probabilities of the large language model for generating the baseline answer results; After giving the retrieval information, the large language model is used to sample the given question and the target document set returned by retrieval to obtain corresponding enhanced answer results and second generation probabilities of the large language model for generating the enhanced answer results; The baseline answer results and the enhanced answer results are judged, and answer results expressing semantic equivalence are merged into the same semantic class set; The first semantic perplexity and the second semantic perplexity of the large language model for the given question before and after giving the retrieval information are determined by calculating the sum of the generation probabilities of all answer results in the corresponding semantic class set based on the first generation probability and the second generation probability respectively; The contribution of the retrieval information is measured by analyzing the gain between the first semantic perplexity and the second semantic perplexity; The large language model is optimized and trained according to the enhanced sample data set constructed by the contribution.

2. The semantic perplexity-based retrieval method of claim 1, wherein, The judgment of the baseline answer results and the enhanced answer results, which merges the target answer results expressing semantic equivalence into the same semantic class set, comprises the following steps: The reference answer of the given question is used as a benchmark to judge the semantic equivalence of the baseline answer results and the enhanced answer results before and after giving the retrieval information; The answer results expressing equivalent semantics with the reference answer are merged into the same semantic class set through clustering analysis of the judgment results.

3. The method of claim 2, wherein the semantic perplexity is calculated by: ###0001### where N is the number of words in the document, and P(w) is the probability of the word w in the document. The judgment of the semantic equivalence of the baseline answer results and the enhanced answer results comprises the following steps: A dynamic threshold judgment mechanism is introduced, and a text entailment model is used to judge the semantic equivalence of the baseline answer results and the enhanced answer results.

4. The method of claim 1, wherein the semantic perplexity-based retrieval is performed by: determining a semantic perplexity of the query; and retrieving the one or more results based on the semantic perplexity. The analysis of the gain between the first semantic perplexity and the second semantic perplexity comprises the following formula: ΔSePer = P M (a*|D) - P M (a*) where ΔSePer is the information gain of the search; P M (a*|D) is the likelihood probability of generating the answer result after the search information is given; P M (a*) is the likelihood probability of generating the answer result before the search information is given; a* is the reference answer.

5. The method of claim 1, wherein the semantic perplexity based retrieval is performed by: determining a semantic perplexity of the query; and retrieving the one or more results based on the semantic perplexity. The analysis of the gain between the first semantic perplexity and the second semantic perplexity to measure the contribution of the retrieval information further comprises the following steps: In the multi-hop reasoning scenario, the semantic perplexity of each retrieval process is calculated independently; The contribution of each retrieval process is determined by the calculation result, and the thinking reasoning chain of the large language model is optimized.

6. The method of claim 1, wherein the semantic perplexity based retrieval is performed by: determining a semantic perplexity of the query; and retrieving the one or more results based on the semantic perplexity. The optimization and training of the large language model according to the enhanced sample data set constructed by the contribution comprises the following steps: The enhanced sample data set is constructed according to the high-contribution retrieval information determined by the measurement result; During the training process, the prompt information and the retrieval content generated by the large language model are optimized according to the enhanced sample data set.

7. The method of claim 1, wherein the semantic perplexity based retrieval is performed by: determining a semantic perplexity of the query; and retrieving the one or more results based on the semantic perplexity. The method further comprises the following steps: When the reference answer contains multiple correct answers, the language perplexity gain corresponding to each correct answer is calculated; The contribution of the multi-answer retrieval information is determined by weighted analysis of the calculation results.

8. A semantic perplexity-based search system, characterized by, The method comprises the following steps: A benchmark sampling module is configured to sample a given question based on a preset large language model to obtain corresponding benchmark answer results and first generation probabilities of the large language model generating each benchmark answer result before retrieval information is given; An enhanced sampling module is configured to sample the given question and a target document set returned by retrieval using the large language model to obtain each enhanced answer result and second generation probabilities of the large language model generating each enhanced answer result after the retrieval information is given; A semantic judgment module is configured to judge the benchmark answer results and the enhanced answer results and merge target answer results expressing semantic equivalence into a same semantic class set; A semantic perplexity analysis module is configured to calculate a sum of generation probabilities of all answer results in the corresponding semantic class set based on the first generation probabilities and the second generation probabilities to determine first and second semantic perplexities of the large language model for the given question before and after the retrieval information is given; A retrieval contribution analysis module is configured to measure a contribution degree of retrieval information by analyzing a gain between the first and second semantic perplexities; A model optimization module is configured to optimize and train the large language model according to an enhanced sample data set constructed by the contribution degree.

9. A computer device, comprising: A computer readable storage medium stores a computer program, wherein a device where the computer readable storage medium is located implements the retrieval method based on semantic perplexity as claimed in any one of claims 1 to 7 when the computer program is executed.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, wherein a device where the computer readable storage medium is located implements the retrieval method based on semantic perplexity as claimed in any one of claims 1 to 7 when the computer program is executed.

Images