Information retrieval method and device, storage medium and equipment

Abstract

The invention discloses an information retrieval method and device, a storage medium and equipment. The method comprises the steps that firstly, a to-be-retrieved target text input by a target user is acquired; then performing preliminary retrieval according to the target text to obtain N candidate texts related to the target text; and then, performing correlation evaluation on the N candidate texts by utilizing a preset large language model, pre-screening M candidate texts meeting screening conditions from the N candidate texts according to an evaluation result, re-sorting the M candidate texts, and determining an information retrieval result for the target text according to an obtained sorting result. It can be seen that the pre-screening step is added before reordering, the candidate texts which do not meet the screening condition are eliminated firstly, then the remaining candidate texts are used for reordering, the ordering efficiency and accuracy can be improved, the efficiency and accuracy of information retrieval can be improved, and the retrieval experience of the target user is also improved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to an information retrieval method, apparatus, storage medium and device. Background Art

[0002] With the continuous development of artificial intelligence and information technology, the application scenarios of human-computer interaction are becoming more and more extensive. Various large language models (LLMs) have appeared in people's lives and work, which can provide people with intelligent interaction functions in many application scenarios such as information retrieval to assist users in completing various behavioral intentions.

[0003] At present, when using large language models (LLMs) for information retrieval, it usually only includes a preliminary retrieval stage and a re-ranking stage. The preliminary retrieval stage is to retrieve document paragraphs related to user queries from a large-scale corpus. The re-ranking stage focuses on re-ranking the candidate paragraphs retrieved in the preliminary retrieval stage according to their relevance. However, this information retrieval method is to pass all the retrieved paragraphs in the preliminary retrieval stage directly to the LLM-based rearranger for re-ranking, which may cause the LLM-based rearranger to be misled by irrelevant paragraphs, thereby affecting the accuracy of the ranking results, thereby reducing the accuracy of the information retrieval results and affecting the user's interactive experience. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide an information retrieval method, apparatus, storage medium and device, which improves the efficiency and accuracy of sorting by adding a pre-screening step before re-sorting to exclude irrelevant paragraphs, thereby improving the efficiency and accuracy of information retrieval and further improving the user's interactive experience.

[0005] The present application provides an information retrieval method, including:

[0006] Obtain the target text to be retrieved input by the target user;

[0007] Performing a preliminary search based on the target text to obtain N candidate texts related to the target text, wherein N is a positive integer greater than 0;

[0008] Using a preset large language model, the N candidate texts are evaluated for relevance, and M candidate texts that meet a screening condition are pre-screened from the N candidate texts according to the evaluation result; M is a positive integer greater than 0 and not greater than N;

[0009] The M candidate texts are re-ranked, and an information retrieval result for the target text is determined according to the obtained ranking result.

[0010] In a possible implementation, the using of a preset large language model to perform relevance evaluation on the N candidate texts, and pre-screening M candidate texts that meet a screening condition from the N candidate texts according to the evaluation result, includes:

[0011] Using a preset large language model, the N candidate texts are evaluated for relevance, and a first relevance score corresponding to each of the N candidate texts is determined;

[0012] M candidate texts whose first relevance scores are lower than a preset score threshold are screened out from the N candidate texts as the M candidate texts that meet the screening condition.

[0013] In a possible implementation, the preset score threshold is determined after dynamic adjustment based on existing expert experience and experimental results.

[0014] In a possible implementation, the preset large language model is trained as follows:

[0015] Obtain a first sample text and a second sample text;

[0016] The initial large language model is trained using the first sample text, the second sample text and the target loss function to obtain a large language model with a relevance evaluation function.

[0017] In a possible implementation, the target loss function is a mean square error loss function.

[0018] In a possible implementation, re-ranking the M candidate texts and determining the information retrieval result for the target text according to the obtained ranking result includes:

[0019] The target text and the M candidate texts are combined with prompt instructions and input into a preset large language model, and the relevance of the M candidate texts is evaluated again to determine a second relevance score corresponding to each of the M candidate texts;

[0020] The M candidate texts are reordered according to the magnitude of the second relevance scores, and an information retrieval result for the target text is determined based on the obtained ordering result.

[0021] In a possible implementation, the prompt instruction is determined based on the characteristics of a preset large language model, an evaluation dimension of text relevance, and a prompt template.

[0022] The present application also provides an information retrieval device, including:

[0023] A first acquisition unit, used to acquire a target text to be retrieved input by a target user;

[0024] A preliminary search unit, used to perform preliminary search based on the target text to obtain N candidate texts related to the target text; N is a positive integer greater than 0;

[0025] A pre-screening unit, configured to use a preset large language model to perform relevance evaluation on the N candidate texts, and pre-screen M candidate texts that meet a screening condition from the N candidate texts according to the evaluation result; wherein M is a positive integer greater than 0 and not greater than N;

[0026] The sorting unit is used to re-sort the M candidate texts and determine the information retrieval result for the target text according to the obtained sorting result.

[0027] In a possible implementation, the pre-screening unit includes:

[0028] A first evaluation subunit is used to perform relevance evaluation on the N candidate texts using a preset large language model, and determine a first relevance score corresponding to each of the N candidate texts;

[0029] The screening subunit is used to screen out M candidate texts whose first relevance scores are lower than a preset score threshold from the N candidate texts as the M candidate texts that meet the screening condition.

[0030] In a possible implementation, the preset score threshold is determined after dynamic adjustment based on existing expert experience and experimental results.

[0031] In a possible implementation manner, the device further includes:

[0032] A second acquiring unit, used for acquiring a first sample text and a second sample text;

[0033] The training unit is used to train the initial large language model using the first sample text, the second sample text and the target loss function to obtain a large language model with a relevance evaluation function.

[0034] In a possible implementation, the target loss function is a mean square error loss function.

[0035] In a possible implementation, the sorting unit includes:

[0036] A second evaluation subunit is used to use the target text and the M candidate texts in combination with the prompt instruction, input them into a preset large language model, perform relevance evaluation on the M candidate texts again, and determine a second relevance score corresponding to each candidate text in the M candidate texts;

[0037] The sorting subunit is used to re-sort the M candidate texts according to the size of the second relevance score, and determine the information retrieval result for the target text according to the obtained sorting result.

[0038] In a possible implementation, the prompt instruction is determined based on the characteristics of a preset large language model, an evaluation dimension of text relevance, and a prompt template.

[0039] The embodiment of the present application also provides an information retrieval device, including: a processor, a memory, and a system bus;

[0040] The processor and the memory are connected via the system bus;

[0041] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementations of the above-mentioned information retrieval method.

[0042] An embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementations of the above-mentioned information retrieval method.

[0043] The embodiment of the present application further provides a computer program product. When the computer program product is executed on a terminal device, the terminal device executes any one of the implementation modes of the above-mentioned information retrieval method.

[0044] The information retrieval method, apparatus, storage medium and device provided by the embodiment of the present application first obtain the target text to be retrieved input by the target user; then perform a preliminary search based on the target text to obtain N candidate texts related to the target text; wherein N is a positive integer greater than 0. Next, use a preset large language model to evaluate the relevance of the N candidate texts, and pre-screen M candidate texts that meet the screening conditions from the N candidate texts according to the evaluation results; wherein M is a positive integer greater than 0 and not greater than N; then re-rank the M candidate texts, and determine the information retrieval result for the target text according to the obtained ranking result.

[0045] It can be seen that after the preliminary search is performed based on the target text input by the target user, the present application no longer passes all the retrieved candidate texts directly to the LLM-based rearranger for reordering, but adds a pre-screening step before reordering. The candidate texts that meet the screening conditions are pre-screened from the candidate texts, and then these candidate texts are used for reordering, thereby improving the sorting efficiency and accuracy, which in turn can improve the efficiency and accuracy of information retrieval, and also improve the retrieval experience of the target user. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 A flowchart of an information retrieval method provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of the composition of an information retrieval device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] With the widespread application of large language models (LLMs) in natural language processing tasks, their application in the field of information retrieval is also increasing. At present, when using large language models (LLMs) for information retrieval, it usually only includes the preliminary retrieval stage and the re-ranking stage. Among them, the preliminary retrieval stage is to retrieve document paragraphs related to the user query from a large corpus. The re-ranking stage focuses on re-ranking the candidate paragraphs retrieved in the preliminary retrieval stage according to their relevance.

[0050] However, this information retrieval method is to pass all the retrieved paragraphs in the preliminary retrieval stage directly to the LLM-based rearranger for re-ranking. Too many irrelevant paragraphs may increase the processing difficulty of the rearranger, which will not only cause information overload, but also the interference of irrelevant paragraph information may mislead the rearranger, resulting in an incorrect assessment of the relevance of the paragraphs. In addition, since the paragraphs retrieved in the preliminary retrieval stage lack context, it will also affect the rearranger's accurate judgment of the relevance of each paragraph. In addition, the model bias of the large language model LLM itself may also lead to inaccurate re-ranking of each paragraph. Taking into account the combined effect of the aforementioned factors, it will be difficult for the LLM-based rearranger to perform accurate ranking, which will affect the accuracy of the ranking results, thereby reducing the accuracy of the information retrieval results and affecting the user's information retrieval experience.

[0051] In addition, for users with limited resources, they cannot use the more powerful top-level large language model LLM or closed-source LLM for information retrieval, which will also reduce the accuracy of the retrieval results and affect their retrieval experience.

[0052] In order to solve the above defects, the present application provides an information retrieval method, which first obtains the target text to be retrieved input by the target user; then performs a preliminary search based on the target text to obtain N candidate texts related to the target text; wherein N is a positive integer greater than 0. Next, the N candidate texts are evaluated for relevance using a preset large language model, and M candidate texts that meet the screening conditions are pre-screened from the N candidate texts based on the evaluation results; wherein M is a positive integer greater than 0 and not greater than N; then the M candidate texts are re-ranked, and the information retrieval result for the target text is determined based on the ranking result obtained.

[0053] It can be seen that after the preliminary search is performed based on the target text input by the target user, the present application no longer passes all the retrieved candidate texts directly to the LLM-based rearranger for reordering, but adds a pre-screening step before reordering. The candidate texts that meet the screening conditions are pre-screened from the candidate texts, and then these candidate texts are used for reordering, thereby improving the sorting efficiency and accuracy, which in turn can improve the efficiency and accuracy of information retrieval, and also improve the retrieval experience of the target user.

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] First embodiment

[0056] See also Figure 1 , is a flow chart of an information retrieval method provided in this embodiment, the method comprising the following steps:

[0057] S101: Obtaining a target text to be retrieved input by a target user.

[0058] In this embodiment, any text that needs to retrieve relevant information input by the user into the large language model (LLM) or other intelligent interactive software or equipment is defined as the target text to be retrieved, and the user is defined as the target user. In addition, this embodiment does not limit the language type of the target text. For example, the target text can be Chinese text or English text. This embodiment also does not limit the length of the target text, that is, the target text can be a sentence text (i.e., a collection of individual words) or a chapter text (i.e., a collection of individual sentences). And this embodiment also does not limit the input method of the target user for the target text. For example, the target text can be the text content directly input into the large language model (LLM) by the target user, or by converting the voice query request issued by the target user into text content and then inputting it into the large language model (LLM) as the target text to be retrieved.

[0059] Among them, the large language model LLM can be a language model based on deep learning, which can generate new language expressions such as text, sentences, paragraphs and even articles based on the input text content. The large language model LLM uses a large-scale language data set to train language rules and patterns through autoregressive generation, and can simulate human instructions to generate language expressions (such as text data). Specifically, when the large language model LLM generates new text data, it predicts the possibility of the next language unit based on the previously generated content until the complete text data is generated.

[0060] S102: Perform a preliminary search based on the target text to obtain N candidate texts related to the target text; wherein N is a positive integer greater than 0.

[0061] In this embodiment, after obtaining the target text to be retrieved input by the target user through step S101, the existing or future retriever can be further used to preliminarily retrieve N candidate texts related to the target text from a large-scale corpus by using an inverted index, a vector space model, a machine learning model or a hybrid method, so as to perform the subsequent step S103. Among them, these N candidate texts are only possibly related to the target text, but have not been finely processed and sorted. In addition, this embodiment does not limit the value of N, and it only needs to ensure that it is a positive integer greater than 0.

[0062] Among them, this application does not limit the content and construction method of the large-scale corpus. It should be noted that the large-scale corpus is an important resource in the field of linguistic research and natural language processing. It can contain a large amount of real language materials, as well as useful corpus resources obtained after scientific sampling and processing (analysis and processing), which are used to support various language theories and application research, such as the information retrieval research mentioned in this application.

[0063] S103: Using a preset large language model, perform relevance evaluation on N candidate texts, and pre-screen M candidate texts that meet the screening conditions from the N candidate texts according to the evaluation results; wherein M is a positive integer greater than 0 and not greater than N.

[0064] In this embodiment, after N candidate texts related to the target text are preliminarily retrieved through step S102, in order to improve the retrieval efficiency and accuracy of the target file related information, before re-ranking each candidate text, the preset large language model can be used to evaluate the relevance of the N candidate texts, and M candidate texts that meet the screening conditions can be pre-screened from the N candidate texts according to the evaluation results, so as to perform the subsequent S104 re-ranking to avoid the candidate texts with low relevance or irrelevant to the target text affecting the accuracy of the retrieval results. Among them, this embodiment does not limit the value of M, and only needs to ensure that it is a positive integer greater than 0 and not less than N. In addition, this embodiment does not limit the content included in the screening condition, and can be set according to actual conditions and experience values. For example, it can be set that only candidate texts above a certain threshold can be used as one of the M candidate texts, so that candidate documents with low relevance or irrelevant to the target text can be screened out from the N candidate texts, so as to reduce the data processing burden of the subsequent re-ranking stage.

[0065] Specifically, an optional implementation method is that after initially retrieving N candidate texts related to the target text, the deep learning ability of the preset large language model LLM can be used to first evaluate the relevance of the N candidate texts, and determine the relevance score corresponding to each candidate text (here it is defined as the first relevance score, which can be between 0 and 1). The first relevance score reflects the semantic similarity and relevance between the corresponding candidate text and the target text. For example, assuming that the target file input by the target user is "artificial intelligence", and the content of a candidate text initially retrieved by step S102 is "the application of artificial intelligence in medical imaging diagnosis", and after the relevance evaluation is performed on it using the preset large language model LLM, it is determined that its corresponding first relevance score is 0.9, then the first relevance score indicates that the semantic similarity between the candidate text and the target text is high, the subject of the candidate text and the query subject of the target text both involve artificial intelligence, and both focus on the application of artificial intelligence in specific fields; and, it also indicates that the correlation between the candidate text and the target text is also high, and the content of the candidate text details the application of artificial intelligence in medical imaging diagnosis, which is highly relevant to the purpose of the query. Then, M candidate texts whose first relevance scores are lower than a preset score threshold are screened out from the N candidate texts as the M candidate texts that meet the screening condition.

[0066] Among them, this embodiment does not limit the specific value of the preset score threshold. An optional implementation method is that in order to improve the accuracy of the screening results, it can be dynamically adjusted (i.e., continuously iterated and optimized) according to the existing expert experience and experimental results before final determination. Candidate texts below the preset score threshold (i.e., candidate texts with low relevance to the target text) are regarded as candidate texts irrelevant to the target text, so as to exclude them in subsequent processing steps such as re-ranking.

[0067] In this implementation, an iterative optimization method is used to determine the preset score threshold. Through continuous iteration and optimization, an optimal threshold that can ensure both retrieval efficiency and retrieval accuracy is found as the preset score threshold. The factors considered may include but are not limited to: (1) Expert experience, that is, based on the existing expert judgment criteria for relevance, a threshold range can be preliminarily set. However, it should be noted that expert experience is actually manual experience in engineering practice, and expert experience may be different in different business scenarios. (2) Experimental results, that is, by conducting experiments on the test set, the model retrieval performance under different thresholds is compared, such as accuracy, recall rate and other indicators, to select the best threshold. (3) User feedback, that is, according to the user's actual retrieval experience and feedback, the threshold value is adjusted to better meet user needs.

[0068] Furthermore, it is understandable that the number of preset score thresholds and the specific value range depend on the specific application scenario and business experience. For example, the preset score thresholds may include a high correlation threshold, a medium correlation threshold, and a low correlation threshold. Among them, the value of the high correlation threshold can be 0.8 or higher, which is suitable for scenarios where highly relevant results need to be retrieved, such as academic paper retrieval, professional field knowledge base, etc. The value of the medium correlation threshold can be 0.5-0.7, which is suitable for general information retrieval scenarios, such as web page search, news search, etc. The value of the low correlation threshold can be 0.3-0.5, which is suitable for scenarios where more potentially relevant results need to be retrieved, such as search engine exploration mode, recommendation system, etc.

[0069] In addition, it should be noted that the specific composition and specific retrieval process of the preset large language model mentioned in step S103 can be set according to actual conditions. This embodiment does not limit this. It only needs to ensure that the preset large language model has a strong text relevance evaluation function.

[0070] Next, this embodiment will introduce the construction process of the preset large language model, wherein an optional implementation method is that the construction process of the preset large language model may specifically include: first obtaining two sample texts, and defining them as the first sample text and the second sample text respectively, and then using the one sample text and the second sample text and the target loss function to train the existing large language model (such as some resource-constrained LLMs, etc., also referred to as the initial large language model) to obtain a large language model with a strong text relevance evaluation function.

[0071] Specifically, in this implementation, in order to construct a large language model with a strong text relevance evaluation function, a lot of preparatory work needs to be done in advance. First, a large number of text pairs with different degrees of relevance and no relevance need to be collected as sample texts (i.e., first sample texts and second sample texts) to constitute model training data. For example, the collected relevant paragraph text A and paragraph text B can be used as the first sample text and the second sample text, respectively, to form a pair of sample text pairs. And manually mark the corresponding relevance scores of these sample pairs, such as manually marking the relevance scores of paragraph text A and paragraph text B as the manual marking results. Then, according to these sample text pairs and their corresponding manual marking results and the target loss function, the existing large language model (which can be called the initial large language model) can be trained to generate a large language model with a strong text relevance evaluation function.

[0072] Among them, an optional implementation manner is that the target loss function can be (but is not limited to) a mean square error (MSE) loss function.

[0073] Specifically, when conducting model training, a pair of sample text pairs (i.e., the first sample text and the second sample text) can be extracted from the training data in turn as model input, and the corresponding relevance score can be used as output. An appropriate learning rate can be set, and multiple rounds of model training can be performed. The relevance score obtained in each round of training is compared with the corresponding manual annotation result (i.e., the manually annotated relevance score), and the model parameters are updated according to the difference between the two until the preset conditions are met. For example, the value of the target loss function (such as the MSE loss constraint function) is very small and basically unchanged, then the updating of the model parameters is stopped, the model training is completed, and a well-trained large language model with strong text relevance evaluation function is generated.

[0074] On this basis, after a large language model with strong text relevance evaluation function is generated based on sample text training, the generated large language model can be further verified using verification text. The specific verification process can include the following steps (1)-(3):

[0075] Step (1): Obtain a first verification text and a second verification text.

[0076] In this embodiment, in order to verify the above-mentioned trained large language model with strong text relevance evaluation function, it is first necessary to obtain a large number of text pairs with different degrees of relevance and no relevance as verification text data, such as collecting the two ends of the explanation text of a certain word in the reference book as the first verification text and the second verification text. Among them, the verification text data refers to the text pair data that can be used for model verification. After obtaining these verification text pairs and the score labels corresponding to each verification text pair, the subsequent step (2) can be continued.

[0077] Step (2): Input the verification text pair into the above-mentioned trained large language model with strong text relevance evaluation function to obtain the relevance prediction result corresponding to the verification text pair.

[0078] After obtaining the verification text pair (i.e., the first verification text and the second verification text) through step (1), the verification text pair (i.e., the first verification text and the second verification text) can be further input into the above-mentioned trained large language model with strong text relevance evaluation function to obtain the relevance score prediction result corresponding to the verification text pair (i.e., the first verification text and the second verification text) for executing the subsequent step (3).

[0079] Step (3): When the relevance score prediction result of the verification text pair (i.e., the first verification text and the second verification text) is inconsistent with its corresponding score label (such as the manually annotated relevance score), the verification text pair (i.e., the first verification text and the second verification text) is used as a sample text pair (i.e., the first sample text and the second sample text) again, and the parameters of the above-trained large language model with strong text relevance evaluation function are updated. This helps to improve the prediction accuracy and precision of the corresponding text relevance of the audio-visual language model.

[0080] S104: re-rank the M candidate texts, and determine the information retrieval result for the target text according to the obtained ranking result.

[0081] In this embodiment, after pre-screening M candidate texts that meet the screening conditions through step S103, further, the target question text and each candidate text in the M candidate texts can be integrated into the prompt instruction prompt and input into the preset large language model LLM to utilize the internalization ability of the large language model with a strong text relevance evaluation function. After considering the context content, semantic information, grammatical structure and other factors, each candidate text in the M candidate texts is re-evaluated for relevance, and the relevance score corresponding to each candidate text is determined (here it is defined as the second relevance score, which can be between 0 and 1) as the basis for sorting. Then, each candidate text is re-sorted from high to low according to the size of the corresponding second relevance score, or it can be sorted in other ways or using an existing or future reranker (Reranker), which is not limited in this application, to obtain a sorting result, so that the target user can quickly find the most relevant content to the target text according to the sorting result.

[0082] Among them, this embodiment does not limit the specific construction method of the content contained in the prompt instruction. An optional implementation method is that in order to ensure that the preset large language model LLM can perform in-depth analysis on each candidate text among the M candidate texts and generate a more accurate relevance score (i.e., the second relevance score), the prompt instruction can be determined according to the characteristics of the preset large language model LLM, the evaluation dimension of text relevance and the prompt template, so that the relevance score (i.e., the second relevance score) obtained after the prompt instruction is input into the preset large language model LLM not only considers the semantic similarity between the candidate text and the target text, but also considers the contextual information, grammatical structure and other factors in the candidate text, thereby improving the accuracy of subsequent sorting results and retrieval results.

[0083] In this implementation, when designing prompt instructions, the design steps included therein may specifically include but are not limited to: first, determine the evaluation dimensions, such as semantic similarity, keyword matching, syntactic structure, etc. Then design a prompt template, including information such as target text, candidate text, and evaluation dimensions, such as "Please evaluate the similarity between [target text] and [candidate text] on [evaluation dimension].". Next, generate sample data for training the LLM understanding prompt template. Furthermore, reduce model training and evaluation, adjust the prompt template and evaluation dimensions, and optimize the LLM model. Finally, perform multiple rounds of continuous iterative optimization according to the actual application scenario to generate prompt instructions.

[0084] In this way, by executing the above steps S101-S104, after a preliminary search is performed based on the target text input by the target user, all the N retrieved candidate texts are no longer directly passed to the LLM-based rearranger for reordering, but a pre-screening step is added before reordering, which effectively improves the efficiency and accuracy of information retrieval. Among them, the pre-screening step (see step S103 for details) uses the preset large language model LLM to evaluate the relevance of the N candidate texts, and excludes irrelevant candidate texts by setting a preset score threshold to obtain the remaining M candidate texts. Then, the subsequent reordering step (see step S104 for details) guides the LLM to generate an accurate relevance score (i.e., the second relevance score) for each candidate text in the M candidate texts through the design of reasonable prompt instructions, and sorts them according to the score. As a result, the target user can finally find content related to the target text quickly and accurately in the massive data.

[0085] Next, the overall implementation process of the information retrieval method provided by this application will be described by way of example:

[0086] For example: Assume that the preset score threshold is 0.5, that is, the candidate text with a first relevance score lower than 0.5 (i.e., the candidate text with a low relevance to the target text) is regarded as a candidate text that is not relevant to the target text. In addition, four candidate texts related to the target text are preliminarily retrieved through step S102 and their corresponding first relevance scores are: (1) "XX is a XXX technology widely used in the medical field. It can help doctors perform XXX operations and improve the accuracy of diagnosis." The corresponding first relevance score is 0.8. (2) "XX's applications in the medical field mainly include XXX and XXX. It can help patients perform XXX treatment." The corresponding first relevance score is 0.7. (3) "XX is a technology widely used in the industrial field. It can help engineers perform XXX operations and improve production efficiency." The corresponding first relevance score is 0.3. (4) "XX is an emerging XXX technology. It has broad application prospects in the XXX field." The corresponding first relevance score is 0.3.

[0087] It can be seen that the first relevance scores 0.4 and 0.3 corresponding to the candidate texts (3) and (4) are both less than the preset score threshold 0.5, and will be excluded. On the contrary, the first relevance scores 0.8 and 0.7 corresponding to the candidate texts (1) and (2) are both greater than the preset score threshold 0.5, and will be retained as two of the M candidate texts for the subsequent re-ranking processing step.

[0088] During the re-ranking stage, the quality of the prompts used in this application needs to be designed according to the characteristics of LLM and the retrieval target. Examples of prompt instructions may include but are not limited to: (1) direct prompts, that is, directly inputting the target text content and the candidate text content into LLM, and requiring LLM to generate a relevance score between 0 and 1. (2) Guided prompts, specifically by adding some guide words or sentences to guide LLM to focus on specific information in the candidate text, for example: "Please evaluate the relevance of this content to the following target text to be retrieved: XX application in the medical field. This paragraph mainly describes the application of XX technology in XXX. Please generate a relevance score based on this information." etc. (3) Multi-round prompts, specifically, through multi-round interactions, gradually guide LLM to understand the query intent of the target user inputting the target text and the candidate text content, and generate a more accurate relevance score (i.e., the second relevance score).

[0089] Furthermore, the remaining M candidate texts can be sorted according to the second relevance scores generated by LLM. For example, still based on the above example, assuming that the second relevance scores generated by LLM for the candidate texts (1) and (2) are 0.85 and 0.75 respectively, the sorting results are as follows:

[0090] 1. “XX is a XXX technology that is widely used in the medical field. It can help doctors perform XXX operations and improve the accuracy of diagnosis.” (Relevance score: 0.85)

[0091] 2. “The applications of XX in the medical field mainly include XXX and XXX, which can help patients undergo XXX treatment.” (Relevance score: 0.75).

[0092] On this basis, the target user can determine the information retrieval results for the target text according to the sorting results.

[0093] In summary, the information retrieval method provided in this embodiment first obtains the target text to be retrieved input by the target user; then performs a preliminary search based on the target text to obtain N candidate texts related to the target text; wherein N is a positive integer greater than 0. Next, the N candidate texts are evaluated for relevance using a preset large language model, and M candidate texts that meet the screening conditions are pre-screened from the N candidate texts according to the evaluation results; wherein M is a positive integer greater than 0 and not greater than N; then the M candidate texts are re-ranked, and the information retrieval result for the target text is determined according to the obtained ranking result.

[0094] It can be seen that after the preliminary search is performed based on the target text input by the target user, the present application no longer passes all the retrieved candidate texts directly to the LLM-based rearranger for reordering, but adds a pre-screening step before reordering. The candidate texts that meet the screening conditions are pre-screened from the candidate texts, and then these candidate texts are used for reordering, thereby improving the sorting efficiency and accuracy, which in turn can improve the efficiency and accuracy of information retrieval, and also improve the retrieval experience of the target user.

[0095] Second embodiment

[0096] This embodiment will introduce an information retrieval device. For related content, please refer to the above method embodiment.

[0097] See also Figure 2 , is a schematic diagram of the composition of an information retrieval device provided in this embodiment, the device 200 includes:

[0098] The first acquisition unit 201 is used to acquire the target text to be searched input by the target user;

[0099] A preliminary search unit 202 is used to perform a preliminary search based on the target text to obtain N candidate texts related to the target text; N is a positive integer greater than 0;

[0100] The pre-screening unit 203 is used to use a preset large language model to perform relevance evaluation on the N candidate texts, and pre-screen M candidate texts that meet the screening condition from the N candidate texts according to the evaluation result; M is a positive integer greater than 0 and not greater than N;

[0101] The sorting unit 204 is used to re-sort the M candidate texts and determine the information retrieval result for the target text according to the obtained sorting result.

[0102] In one implementation of this embodiment, the pre-screening unit 203 includes:

[0103] A first evaluation subunit is used to perform relevance evaluation on the N candidate texts using a preset large language model, and determine a first relevance score corresponding to each of the N candidate texts;

[0104] The screening subunit is used to screen out M candidate texts whose first relevance scores are lower than a preset score threshold from the N candidate texts as the M candidate texts that meet the screening condition.

[0105] In an implementation of this embodiment, the preset score threshold is determined after dynamic adjustment based on existing expert experience and experimental results.

[0106] In an implementation of this embodiment, the device further includes:

[0107] A second acquiring unit, used for acquiring a first sample text and a second sample text;

[0108] The training unit is used to train the initial large language model using the first sample text, the second sample text and the target loss function to obtain a large language model with a relevance evaluation function.

[0109] In an implementation of this embodiment, the target loss function is a mean square error loss function.

[0110] In an implementation of this embodiment, the sorting unit 204 includes:

[0111] A second evaluation subunit is used to use the target text and the M candidate texts in combination with the prompt instruction, input them into a preset large language model, perform relevance evaluation on the M candidate texts again, and determine a second relevance score corresponding to each candidate text in the M candidate texts;

[0112] The sorting subunit is used to re-sort the M candidate texts according to the size of the second relevance score, and determine the information retrieval result for the target text according to the obtained sorting result.

[0113] In one implementation of this embodiment, the prompt instruction is determined based on the characteristics of a preset large language model, the evaluation dimension of text relevance, and the prompt template.

[0114] Furthermore, an embodiment of the present application also provides an information retrieval device, including: a processor, a memory, and a system bus;

[0115] The processor and the memory are connected via the system bus;

[0116] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementation methods of the above-mentioned information retrieval method.

[0117] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned information retrieval method.

[0118] Furthermore, an embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the implementation methods of the above-mentioned information retrieval method.

[0119] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0120] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0121] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0122] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An information retrieval method, characterized in that: include: Obtain the target text to be retrieved input by the target user; Performing a preliminary search based on the target text to obtain N candidate texts related to the target text, wherein N is a positive integer greater than 0; Using a preset large language model, the N candidate texts are evaluated for relevance, and M candidate texts that meet a screening condition are pre-screened from the N candidate texts according to the evaluation result; M is a positive integer greater than 0 and not greater than N; The M candidate texts are re-ranked, and an information retrieval result for the target text is determined according to the obtained ranking result.

2. The method according to claim 1, characterized in that The method of using a preset large language model to evaluate the relevance of the N candidate texts, and pre-screening M candidate texts that meet the screening conditions from the N candidate texts according to the evaluation results, includes: Using a preset large language model, the N candidate texts are evaluated for relevance, and a first relevance score corresponding to each of the N candidate texts is determined; M candidate texts whose first relevance scores are lower than a preset score threshold are screened out from the N candidate texts as the M candidate texts that meet the screening condition.

3. The method according to claim 2, characterized in that The preset score threshold is determined after dynamic adjustment based on existing expert experience and experimental results.

4. The method according to claim 1, characterized in that: The training method of the preset large language model is as follows: Obtain a first sample text and a second sample text; The initial large language model is trained using the first sample text, the second sample text and the target loss function to obtain a large language model with a relevance evaluation function.

5. The method according to claim 4, characterized in that The objective loss function is a mean square error loss function.

6. The method according to claim 1, characterized in that The reordering of the M candidate texts and determining the information retrieval result for the target text according to the obtained ordering result includes: The target text and the M candidate texts are combined with prompt instructions and input into a preset large language model, and the relevance of the M candidate texts is evaluated again to determine a second relevance score corresponding to each of the M candidate texts; The M candidate texts are reordered according to the magnitude of the second relevance scores, and an information retrieval result for the target text is determined based on the obtained ordering result.

7. The method according to claim 6, characterized in that The prompt instruction is determined based on the characteristics of a preset large language model, the evaluation dimension of text relevance and the prompt template.

8. An information retrieval device, characterized in that: include: A first acquisition unit, used to acquire a target text to be retrieved input by a target user; A preliminary search unit, used to perform preliminary search based on the target text to obtain N candidate texts related to the target text; N is a positive integer greater than 0; A pre-screening unit, configured to use a preset large language model to perform relevance evaluation on the N candidate texts, and pre-screen M candidate texts that meet a screening condition from the N candidate texts according to the evaluation result; wherein M is a positive integer greater than 0 and not greater than N; The sorting unit is used to re-sort the M candidate texts and determine the information retrieval result for the target text according to the obtained sorting result.

9. An information retrieval device, characterized in that: include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 7.

Images