Retrieval method and device

Abstract

The invention provides a retrieval method and device, the method can be applied to a large language model, and the method comprises the following steps: obtaining to-be-retrieved information; generating a first retrieval formula according to the to-be-retrieved information; obtaining a first retrieval result and an evaluation result, and generating a second retrieval formula according to the to-be-retrieved information, the first retrieval result and the evaluation result; the first retrieval result is obtained by a retrieval system through retrieval according to the first retrieval formula, the evaluation result is obtained by an evaluation system according to the first retrieval result, and the second retrieval formula is used for obtaining a next round of retrieval result by the retrieval system.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular, to a retrieval method and device. Background Art

[0002] The search formula is an important part of novelty search in science and technology, which directly affects the efficiency of retrieval and the accuracy of novelty search results. In the search formula, the search terms need to accurately reflect the technical theme of the novelty search project. However, due to the diversity of professional terms, the same concept may have multiple synonyms or near-synonyms, making it difficult to accurately and completely determine the search terms. Moreover, formulating the search formula also requires an understanding of the construction method of the search formula. These problems all lead to inaccurate search formulas or ineffective determination of search formulas, and further result in poor retrieval efficiency and results. Summary of the Invention

[0003] The present disclosure provides a retrieval method, device, equipment, storage medium, and retrieval platform.

[0004] According to a first aspect of the present disclosure, there is provided a retrieval method applied to a large language model, the method including:

[0005] Obtain information to be retrieved;

[0006] Generate a first search formula according to the information to be retrieved;

[0007] Obtain a first retrieval result and an evaluation result, and generate a second search formula according to the information to be retrieved, the first retrieval result, and the evaluation result; the first retrieval result is retrieved by a retrieval system according to the first search formula, the evaluation result is obtained by an evaluation system according to the first retrieval result, and the second search formula is used for the retrieval system to obtain the next round of retrieval results.

[0008] According to a second aspect of the present disclosure, there is provided a retrieval device, the device including:

[0009] An obtaining module for obtaining information to be retrieved;

[0010] A first processing module for generating a first search formula according to the information to be retrieved;

[0011] The obtaining module is further configured to obtain a first retrieval result and an evaluation result;

[0012] The first processing module is further configured to generate a second search formula according to the information to be retrieved, the first retrieval result, and the evaluation result; the first retrieval result is retrieved by a retrieval system according to the first search formula, the evaluation result is obtained by an evaluation system according to the first retrieval result, and the second search formula is used for the retrieval system to obtain the next round of retrieval results.

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in the present disclosure.

[0017] According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method described in the present disclosure.

[0018] According to a fifth aspect of the present disclosure, there is provided a retrieval method, which is applied to a retrieval platform, and the retrieval platform includes: a large language model, a retrieval system, and an evaluation system; the method includes:

[0019] The large language model obtains the information to be retrieved and determines a first retrieval formula according to the information to be retrieved;

[0020] The retrieval system performs a retrieval according to the first retrieval formula to obtain a first retrieval result;

[0021] The evaluation system evaluates the first retrieval result to obtain an evaluation result.

[0022] According to a sixth aspect of the present disclosure, there is provided a retrieval platform, which includes: a large language model, a retrieval system, and an evaluation system;

[0023] The large language model is configured to obtain the information to be retrieved and determine a first retrieval formula according to the information to be retrieved;

[0024] The retrieval system is configured to perform a retrieval according to the first retrieval formula to obtain a first retrieval result;

[0025] The evaluation system is configured to evaluate the first retrieval result to obtain an evaluation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features, and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the drawings, several embodiments of the present disclosure are shown by way of illustration and not limitation, wherein:

[0027] In the drawings, the same or corresponding reference numerals denote the same or corresponding parts.

[0028] Figure 1 Flow chart of a retrieval method provided by an embodiment of the present disclosure;

[0029] Figure 2 Flow chart of another retrieval method provided by an embodiment of the present disclosure;

[0030] Figure 3 Flow chart of a retrieval method provided by an application embodiment of the present disclosure;

[0031] Figure 4 Flow chart of another retrieval method provided by an application embodiment of the present disclosure;

[0032] Figure 5 Schematic diagram of a retrieval device provided by an embodiment of the present disclosure;

[0033] Figure 6 Schematic diagram of the structure of a retrieval platform provided by an embodiment of the present disclosure;

[0034] Figure 7 Schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Detailed implementation manners

[0035] To make the objectives, features, and advantages of the present disclosure more obvious and understandable, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative efforts fall within the protection scope of the present disclosure.

[0036] In the following description, "some embodiments" are involved, which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0037] In the following description, the terms "first / second" only distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first / second" can be interchanged with a specific order or sequence when allowed, so that the embodiments of the present disclosure described here can be implemented in an order other than that illustrated or described here.

[0038] Unless otherwise defined, all technical and scientific terms used in the present disclosure have the same meaning as commonly understood by those skilled in the technical field to which the present disclosure belongs. The terms used in the present disclosure are only for the purpose of describing the embodiments of the present disclosure and are not intended to limit the present disclosure.

[0039] It should be understood that in various embodiments of the present disclosure, the magnitudes of the sequence numbers of the various implementation processes do not imply the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

[0040] Figure 1 FIG. is a schematic flow chart of a retrieval method provided by an embodiment of the present disclosure; as Figure 1 shown, the method is applied to a large language model (LLM, Large Language Model), and the method includes:

[0041] Step 101, obtain the information to be retrieved;

[0042] Step 102, generate a first retrieval formula according to the information to be retrieved;

[0043] Step 103, obtain a first retrieval result and an evaluation result, and generate a second retrieval formula according to the information to be retrieved, the first retrieval result, and the evaluation result; the first retrieval result is retrieved by a retrieval system according to the first retrieval formula, the evaluation result is obtained by an evaluation system according to the first retrieval result, and the second retrieval formula is used for the retrieval system to obtain the next round of retrieval results.

[0044] Here, the information to be retrieved may include a text describing a content to be retrieved, and the text may be a set of various keywords or key information. It may be a descriptive text, a retrieval formula, a combination of one or more keywords, a combination of one or more retrieval items, etc.

[0045] Among them, a retrieval formula refers to a query string understood and operated by a search engine, which is composed of keywords, logical operators, and search instructions (search syntax), etc. Keywords are the main body of the retrieval formula, while logical operators and search instructions limit the search for keywords from different angles according to specific query requirements.

[0046] Keywords are the main body of the retrieval formula and are used to describe the information content that the user wants to find. If the information to be retrieved is a descriptive text, the keywords may be the key words extracted from the information to be retrieved and related words of the key words (such as translations, synonyms, etc.); if the information to be retrieved is a combination of several retrieval words or retrieval items, the words in the retrieval words or retrieval items are used as keywords; if the information to be retrieved is a retrieval formula, the words in the retrieval formula are the keywords.

[0047] Logical operators: used to connect multiple keywords and represent the relationship between them. Commonly used logical operators include: AND or "*": indicating that two retrieval words must appear simultaneously; OR or "+": indicating that at least one of the two retrieval words appears; NOT or "-": indicating excluding a certain retrieval word.

[0048] Search instructions: Used to further limit the search scope, such as "su = subject", "ti = title", "ky = keyword", etc.

[0049] For example, the retrieval formula can be in the following format: (AA AND BB) OR (CC AND DD), where (AA AND BB) and (CC AND DD) are both retrieval items, and AA, BB, CC, and DD are all retrieval words, that is, keywords. AND and OR are logical operators indicating the connection relationship between retrieval words or retrieval items.

[0050] For example, an example of a retrieval formula in the above format can be expressed as: (intelligent cabinet AND intelligent storage cabinet) AND (dangerous goods AND hazardous substances); it should be noted that the above is only a simple example. In fact, there can be more retrieval words and / or retrieval items, or other different logical operators can be used. For another example, another example of a retrieval formula containing the above format can be expressed as: (intelligent cabinet AND intelligent storage cabinet) AND (dangerous goods OR hazardous substances) NOT (dangerous chemicals AND dangerous reagents).

[0051] The content to be retrieved can be stored in various information carriers, such as databases, file libraries, network storage devices, etc. For example, there is an academic literature database that includes all the included journal papers, dissertations, research reports, and other literature contents. Users can query the database through the information to be retrieved and obtain the corresponding papers, reports, etc.

[0052] If the evaluation result of the first retrieval result does not meet the target conditions, a retrieval formula update operation can be performed, that is, a new second retrieval formula is generated, so that the retrieval system can continue to retrieve according to the second retrieval formula. In this way, the large language model interacts with the information to be retrieved and the environment (that is, the first retrieval result and the evaluation result) to learn how to construct a more effective retrieval formula, greatly improving the efficiency and accuracy of novelty search, and at the same time reducing the workload of novelty search personnel.

[0053] In some embodiments, the method further includes:

[0054] If the evaluation result does not meet the target conditions, perform at least one round of retrieval formula update operation until a retrieval result whose evaluation result meets the target conditions is obtained; each round of the retrieval formula update operation is used to update the retrieval formula using the large language model; the updated retrieval formula is used for the retrieval system to retrieve again.

[0055] Here, the evaluation result is obtained by the evaluation system based on the first retrieval result; the first retrieval result is retrieved by the retrieval system according to the first retrieval formula; that is, the retrieval system can retrieve according to the retrieval formula generated by the large language model to obtain the corresponding retrieval result, such as the above first retrieval formula and the corresponding first retrieval result; and the evaluation result can evaluate the retrieval result to obtain the evaluation result, such as evaluating the first retrieval result to obtain the corresponding evaluation result; if the evaluation result does not meet the target condition, a retrieval formula update operation can be performed, and a new retrieval formula can be obtained through the update operation, so that the retrieval system can continue to retrieve according to the updated retrieval formula, and so on, until an evaluation result that meets the target condition is obtained, that is, a retrieval result that meets the requirements is obtained.

[0056] Here, performing the retrieval formula update operation obtains the updated retrieval formula, and the content of the updated retrieval formula that constitutes the retrieval formula is at least partially different from that of the previous round of retrieval formula; for example, the content of the first retrieval formula and the second retrieval formula that constitutes the retrieval formula is at least partially different.

[0057] Provide an example. The first retrieval formula is assumed to be: (intelligent cabinet AND intelligent storage cabinet) AND (dangerous goods OR dangerous substances);

[0058] The second retrieval formula obtained after update needs to be different from the first retrieval formula. For example, it can be: (intelligent cabinet AND intelligent storage cabinet AND intelligent storage cabinet) AND (dangerous goods OR dangerous substances); it adds "intelligent storage cabinet";

[0059] Or, the second retrieval formula obtained after update is: (intelligent cabinet AND intelligent storage cabinet) AND (dangerous goods OR dangerous substances) NOT (hazardous chemicals AND dangerous reagents); it adds the retrieval item "NOT (hazardous chemicals AND dangerous reagents)".

[0060] Combined with the above examples, it can be seen that the i-th retrieval formula is different from the i + 1-th retrieval formula. Of course, the above is only one example and is not a limitation on the composition or format of the retrieval formula.

[0061] In this way, through the interaction of the large language model based on the information to be retrieved and the environment (that is, the retrieval result and the evaluation result), the retrieval formula is automatically adjusted to construct a more effective retrieval formula, greatly improving the efficiency and accuracy of novelty search, and at the same time reducing the work burden of novelty search personnel.

[0062] In some embodiments, the method further includes:

[0063] Determine the relevant thesaurus of each retrieval term in the retrieval formula used in the previous round;

[0064] Perform at least one round of retrieval formula update operation, including:

[0065] Update the search formula according to the information to be retrieved, the search formula, search results, evaluation results used in the previous round, and the relevant thesauruses of each search term in the search formula.

[0066] Here, the search formula used in the previous round can be in the following format: (AA AND BB) OR (CC AND DD), where (AA AND BB) and (CC AND DD) are both search items, and AA, BB, CC, and DD are all search terms; the above is only a simple example, and there can actually be more search terms and / or search items.

[0067] Updating the search formula can include at least one of the following methods: adding search terms, reducing search terms, replacing search terms, adjusting the search logic, adding search items, reducing search items, etc.

[0068] The search results can be a set, including: each retrieved search document and the document metadata of each search document, such as: document title, document abstract, document keyword list, etc.

[0069] The evaluation result can be an evaluation of the search results, such as including: the number of documents with a relevance score exceeding the threshold, the average relevance score, the search quality score, the proportion of accepted documents, etc.

[0070] The relevant thesauruses of each search term refer to other words related to the search term, such as synonyms, abbreviations in other language expressions, etc. The relevant thesauruses of each search term can be used to adjust the search term to achieve the update of the search formula. For example, if the search term is intelligent storage cabinet, its relevant thesauruses can include: intelligent storage locker; if the search term is dangerous chemical agent, its relevant thesauruses can include: dangerous chemical substance, dangerous reagent, etc.

[0071] The search formula for the first round can be obtained according to the information to be retrieved. If the information to be retrieved is a text description, the large language model can generate the search formula for the first round according to the information to be retrieved. If the information to be retrieved is a search formula in the above format, this search formula can be directly used as the search formula for the first round.

[0072] When generating the search formula for the first round, since there are no search results and evaluation results obtained from the search formula of the previous round, the search results and evaluation results of the previous round may not be included; for the update operations starting from the second round, the search formula, search results, evaluation results of the previous round, and the relevant thesauruses of each search term in the search formula can be combined to update the search formula.

[0073] The large language model can adopt any model with the above functions or can be obtained through self-training. For example, the method further includes: obtaining at least one first sample data; each of the first sample data includes sample retrieval information and a retrieval formula corresponding to the sample retrieval information; using the at least one first sample data to train a neural network model to obtain the large language model; wherein the sample retrieval information includes at least one of the following: novelty search information, retrieval formula, retrieval result, evaluation result, and relevant thesaurus of each retrieval term in the retrieval formula. Thus, a large language model for generating or adjusting a retrieval formula is obtained.

[0074] In this way, the retrieval formula is continuously updated by the large language model to obtain a retrieval formula that can retrieve more effectively, greatly improving the efficiency and accuracy of novelty search, and at the same time reducing the work burden of novelty search personnel.

[0075] Figure 2 It is a schematic flowchart of another retrieval method provided by an embodiment of the present disclosure; as Figure 2 shown, the retrieval method is applied to a retrieval platform, and the retrieval platform includes: a large language model, a retrieval system, and an evaluation system; the method includes:

[0076] Step 201, the large language model obtains the information to be retrieved and determines a first retrieval formula according to the information to be retrieved;

[0077] Step 202, the retrieval system performs a retrieval according to the first retrieval formula to obtain a first retrieval result;

[0078] Step 203, the evaluation system evaluates the first retrieval result to obtain an evaluation result.

[0079] In some embodiments, the method further includes:

[0080] If the evaluation result does not meet the target condition, at least one round of retrieval formula update operation is executed until a retrieval result whose evaluation result meets the target condition is obtained; each round of the retrieval formula update operation is used to update the retrieval formula by using the large language model; the updated retrieval formula is used for the retrieval system to perform a retrieval again.

[0081] Here, the evaluation result is obtained by the evaluation system based on the first retrieval result; the first retrieval result is retrieved by the retrieval system according to the first retrieval formula; that is, the retrieval system can retrieve according to the retrieval formula generated by the large language model to obtain the corresponding retrieval result, such as the above-mentioned first retrieval formula and the corresponding first retrieval result; and the evaluation result can evaluate the retrieval result to obtain the evaluation result, such as evaluating the first retrieval result to obtain the corresponding evaluation result; if the evaluation result does not meet the target condition, a retrieval formula update operation can be performed, and a new retrieval formula can be obtained through the update operation, so that the retrieval system can continue to retrieve according to the updated retrieval formula, and so on, until an evaluation result that meets the target condition is obtained, that is, a retrieval result that meets the requirements is obtained.

[0082] The method further includes: determining the relevant thesaurus of each retrieval term in the retrieval formula used in the previous round;

[0083] The execution of at least one round of retrieval formula update operation includes:

[0084] The large language model updates the retrieval formula according to the information to be retrieved, the retrieval formula, retrieval result, evaluation result used in the previous round, and the relevant thesaurus of each retrieval term in the retrieval formula.

[0085] For the execution of at least one round of retrieval formula update operation, reference can be specifically made to Figure 1 the description of the method shown. It will not be elaborated here.

[0086] In this way, through the interaction of the large language model according to the information to be retrieved and the environment (that is, the retrieval result and the evaluation result), the retrieval formula is automatically adjusted to construct a more effective retrieval formula, greatly improving the efficiency and accuracy of novelty search, and at the same time reducing the workload of novelty search personnel.

[0087] In some embodiments, the evaluation of the first retrieval result includes at least one of the following:

[0088] Using a first model to identify the bibliographic metadata of each retrieved document in the information to be retrieved and the first retrieval result, and obtaining the relevance score of each retrieved document; according to the relevance score of each retrieved document, determining the number of documents whose relevance score exceeds the threshold and / or the average relevance score;

[0089] Using a second model to identify the document content of each retrieved document in the information to be retrieved and the first retrieval result, and obtaining the content relevance score of each retrieved document; according to the content relevance score of each retrieved document, determining the retrieval quality score;

[0090] Receiving the adoption feedback from the user for the first retrieval result, and determining the proportion of accepted documents according to the adoption feedback.

[0091] Here, for the retrieval results obtained in each round of retrieval formula, at least one of the number of documents with a relevance score exceeding the threshold, the average relevance score, the retrieval quality score, and the proportion of accepted documents can be evaluated.

[0092] The following takes the first retrieval result as an example for illustration. It should be understood that for the subsequently updated retrieval formula, after obtaining the new retrieval result, the same method can also be used for evaluation.

[0093] The evaluation system can be evaluated using at least one model or module, which can include:

[0094] Use the first model to identify the document metadata of each retrieved document in the information to be retrieved and the first retrieval result, and obtain the relevance score of each retrieved document; after obtaining the relevance score, determine the number of documents with a relevance score exceeding the threshold and / or the average relevance score according to the relevance score of each retrieved document.

[0095] Among them, the first model can be an intelligent model for data processing and relevance analysis, using specific algorithms and logical architectures, and having the ability to deeply analyze text. The first model can be obtained based on training or can adopt an existing model with a relevance scoring function. If training is required, it can be trained through a pre-set data set or according to rule settings, so that it can accurately process different types of text and obtain the corresponding analysis objectives. Here, the relevance score can be a score representing the degree of relevance. For example, it is a score between 0 and 1. The closer it is to 1, the more relevant the information to be retrieved is to the retrieved document. On the contrary, the closer it is to 0, the less relevant the information to be retrieved is to the retrieved document. After obtaining the score, the number of documents with a relevance score exceeding the threshold in the current retrieval result can be determined, and / or the average relevance score can be obtained by taking the average of the relevance scores of each retrieved document in the current retrieval result. If the relevance score is a score between 0 and 1, the average relevance score should also be a score between 0 and 1, and the threshold should also be a certain data between 0 and 1, such as 0.6, 0.7, etc. Of course, other numerical ranges can also be used, such as 1-10, 1-100, etc.

[0096] The relevance score is a score generated by the first model for each retrieved document to represent its degree of relevance to the information to be retrieved after identifying and analyzing the document metadata of the information to be retrieved and the retrieved document. The threshold is a pre-set cut-off line used to distinguish documents with higher and lower relevance.

[0097] The number of documents with a relevance score exceeding the threshold refers to the number of documents among all retrieved documents whose relevance score is higher than a certain set threshold. This number can help users intuitively understand how many documents in the retrieval results are highly relevant to the information to be retrieved. For example, in a retrieval on "the application of artificial intelligence in the medical field", the relevance scores of 10 retrieved documents are obtained through the first model, and the set threshold is 60 points (with a full score of 100 points). If the relevance scores of 6 of these documents exceed 60 points, then the number of documents with a relevance score exceeding the threshold is 6. The number of documents with a relevance score exceeding the threshold is a way to measure the effectiveness of the retrieval results. Users can judge whether the retrieval results provide enough highly relevant documents based on this number. If this number is small, it may be considered necessary to adjust the retrieval formula.

[0098] The average relevance score is the average of the relevance scores of all retrieved documents. It is obtained by adding up the relevance scores of all retrieved documents and then dividing by the total number of retrieved documents. This score can reflect the overall degree of relevance between the retrieved documents and the information to be retrieved. For example, taking the 10 retrieved documents mentioned above, their relevance scores are 70, 50, 80, 40, 75, 60, 90, 30, 70, 85 respectively. Adding these scores together gives a total of 650, and then dividing by the total number of documents 10, the average relevance score is 65 points. The average relevance score can be used to evaluate the quality of the retrieval. If the average relevance score is high, it indicates that the retrieval results generally meet the user's needs; if it is low, it may be necessary to further optimize the retrieval formula to improve the overall relevance between the retrieval results and the information to be retrieved.

[0099] The evaluation system uses at least one model or module for evaluation, and may also include:

[0100] Using the second model to identify the document content of each retrieved document in the information to be retrieved and the first retrieval result, obtaining the content relevance score of each retrieved document; determining the retrieval quality score according to the content relevance score of each retrieved document.

[0101] Among them, the second model is also an intelligent model with professional text recognition capabilities. Using specific algorithms and logical architectures, it has the ability to deeply analyze text. The second model can be obtained through training or can use an existing model with a relevance scoring function. If training is required, it can be trained through a pre-prepared dataset or according to rule settings, so that it can accurately process different types of text and achieve the corresponding analysis goals.

[0102] The content relevance score is a quantitative indicator for measuring the degree of association between the document content of the retrieved literature and the information to be retrieved. It is a score obtained by analyzing the document content through a second model. For example, if the information to be retrieved is "the latest progress in electric vehicle battery technology", and the document content of a certain retrieved literature is mainly about the research and development of new materials for electric vehicle batteries, including the detailed test process and results of the new material properties, and mentions the latest technological breakthroughs many times, then the content relevance score of this literature may be relatively high. On the contrary, if a literature mainly focuses on the exterior design of electric vehicles and hardly involves content related to battery technology, the content relevance score will be relatively low. The content relevance score can help users quickly screen out the literature that is closely related to their needs in terms of content and improve the efficiency of information screening.

[0103] The retrieval quality score is a quantitative indicator for comprehensively evaluating the quality of retrieval results. It is an overall score calculated in a certain way (such as weighted average, considering the proportion of high-score literatures, etc.) based on the content relevance scores of each retrieved literature, and is used to judge whether the retrieval process effectively finds the literature highly relevant to the information to be retrieved. For example, in a round of retrieval, if the content relevance scores of most retrieved literatures are relatively high, it indicates that the corresponding retrieval formula has found the literatures that meet the user's needs, and the retrieval quality score will be relatively high. For example, in a retrieval result containing 10 literatures, if the content relevance scores of 8 literatures are above 80 points (out of 100), the retrieval quality score may be relatively high. On the contrary, if the content relevance scores of most literatures are very low, then the retrieval quality score will be low. Through the retrieval quality score, the overall quality of the retrieval results can be intuitively understood. The retrieval quality score can be used as a feedback indicator to improve the retrieval formula to enhance the accuracy and effectiveness of retrieval.

[0104] The evaluation system uses at least one model or module for evaluation, and may also include:

[0105] Receiving the adoption feedback from the user for the first retrieval result and determining the proportion of accepted literatures according to the adoption feedback.

[0106] Here, the first search result can be presented to the user. After the user views the first search result, the user can provide feedback on whether the retrieved documents in the search result meet their needs. For example, the feedback can be a clear "adopt" or "not adopt", and the user can select by clicking the corresponding button. After receiving the user's adoption feedback, the system will calculate the proportion of the number of documents adopted by the user to the total number of retrieved documents. For example, if there are 10 documents in the first search result and the user adopts 4 of them, then the proportion of adopted documents is 40%. The proportion of adopted documents is an evaluation indicator. If the proportion of adopted documents is high, it indicates that the search results obtained by the search formula are more in line with the user's needs, and the search formula may be effective; on the contrary, if the proportion is very low, the search formula needs to be adjusted to improve the user's satisfaction with the search results.

[0107] In this way, through the above evaluation methods, an evaluation result can be obtained that can reflect whether the search formula is effective (i.e., whether it needs to be adjusted).

[0108] In some embodiments, the evaluation result meets the target condition, including at least one of the following:

[0109] The number of documents with a relevance score exceeding the threshold exceeds the first threshold;

[0110] The average relevance score exceeds the second threshold;

[0111] The retrieval quality score exceeds the third threshold;

[0112] The proportion of adopted documents exceeds the fourth threshold.

[0113] Here, the first threshold, the second threshold, the third threshold, and the fourth threshold can be set according to actual application requirements and can also be adjusted by developers or searchers, and there are no restrictions on the values.

[0114] For example, the first threshold is the threshold for the number of documents with a relevance score exceeding the threshold. If you want to quickly obtain a large number of highly relevant documents, the first threshold may be set relatively high. For example, if the scoring range is 0-1, the first threshold can be 0.7, 0.8, etc. to obtain documents with a relatively high relevance score.

[0115] The second threshold is for the average relevance score. If the quality requirements for the search results are very high, the second threshold can be set at around 70-80 points (out of 100 full marks) to ensure that the search results are highly relevant to the information to be retrieved as a whole.

[0116] The third threshold is for the retrieval quality score. If you want the search results to be more accurate, complete, and highly relevant, etc., the third threshold can be set between 80-90 points (out of 100 full marks) to ensure the high quality of the search results.

[0117] The fourth threshold is the proportion of accepted documents. If it is desired that the user is satisfied with the search results, the fourth threshold can be set relatively high, such as 60%-70%, which means that the search formula needs to be able to obtain a sufficient number of documents recognized by the user.

[0118] In some embodiments, evaluating the first search result further includes:

[0119] Performing a weighted calculation based on the number of documents with a relevance score exceeding the threshold, the average relevance score, the search quality score, and / or the proportion of accepted documents to obtain a weighted calculation result;

[0120] Wherein, the evaluation result meeting the target condition further includes: the weighted calculation result exceeding the fifth threshold.

[0121] Here, the calculation method can be expressed by the following formula:

[0122] reward = f (number of documents with a relevance score exceeding the threshold, average relevance score, search quality evaluation, proportion of accepted documents);

[0123] Wherein, reward represents the result obtained through the calculation of function f. If function f represents a weighted calculation, reward represents the weighted calculation result.

[0124] Function f is the result calculated based on factors such as the number of documents, average relevance score, search quality evaluation, and proportion of accepted documents. That is to say, the value of reward depends on these several search-related elements behind, and function f defines the specific calculation logic and relationship between them.

[0125] The weights of the number of documents with a relevance score exceeding the threshold, average relevance score, search quality evaluation, and proportion of accepted documents can be preset according to application requirements, and there are no restrictions on the values.

[0126] Provide an example. The number of documents with a relevance score exceeding the threshold is 5, and the weight is 0.3; the average relevance score is 70 (out of 100), and the weight is 0.3; the search quality score is 75 (out of 100), and the weight is 0.2; the proportion of accepted documents is 60%, and the weight is 0.2. Perform a weighted calculation based on the above values.

[0127] Here, considering these different types of data such as the number of documents with a relevance score exceeding the threshold, average relevance score, search quality score, and proportion of accepted documents, some data are values between 0-100, and some data are values between 0-100%. To unify the standard, all data can be converted into values between 0-1 or other suitable unified ranges, and then a weighted calculation is performed.

[0128] The fifth threshold is a preset criterion used to determine whether the weighted calculation result has reached a satisfactory comprehensive evaluation level. Similar to the first threshold, second threshold, third threshold, and fourth threshold mentioned above, it is set for the comprehensive evaluation result of the weighted calculation.

[0129] If the overall quality requirement for the retrieval results is very high and it is desired that all factors can be satisfied to a certain extent, the fifth threshold may be set relatively high. For example, the fifth threshold can be set around 0.7 - 0.8 (assuming the value range of the weighted calculation result is 0 - 1) to ensure that the retrieval results perform well in multiple aspects such as the quantity of relevance, overall relevance, retrieval quality, and user acceptance.

[0130] In some embodiments, the method further includes one of the following:

[0131] Present at least one round of retrieval expressions and the retrieval results corresponding to each round of the retrieval expressions, and select the target retrieval expression and target retrieval results according to user feedback;

[0132] Present a retrieval log, where the retrieval log includes: at least one round of retrieval expressions and the retrieval results corresponding to each round of the retrieval expressions.

[0133] Here, the retrieval platform provides an interaction function and an interaction interface, and through the interaction function and the interaction interface, at least one of the following interaction methods is implemented.

[0134] First, the retrieval expressions and retrieval results of each round can be refreshed on the page, and a pause button for interaction is provided for the user to choose whether to pause. For example, if the user is satisfied with the retrieval results of the current round, the user can click the pause button, and the user can obtain the retrieval expression, retrieval results of this round, and of course, the corresponding evaluation results, etc. In this way, the user can see the changes in the retrieval expressions and retrieval results in real time.

[0135] Second, the retrieval results and retrieval expressions of each round are presented to the user in the form of a log, and the user can view the log and then select the specific retrieval results to adopt; in this way, the user can see the changes in the retrieval expressions and retrieval results in the log.

[0136] In this way, the interaction function is provided, enabling the user to have the possibility of independently selecting retrieval results, and improving flexibility and user experience.

[0137] The method in the embodiments of the present disclosure models the retrieval strategy optimization problem as a reinforcement learning problem. Among them, the large language model (i.e., the retrieval expression generation model) automatically adjusts the retrieval strategy through interaction with the environment (i.e., the retrieval system and the evaluation system), and iteratively updates the retrieval expressions (specifically, more effective retrieval expressions can be generated by adding, deleting, modifying retrieval terms, etc.).

[0138] The method in the embodiments of the present disclosure can be applied to the answer retrieval scenario or the novelty search scenario. Compared with the manual novelty search method, there is no need for the retrieval personnel or novelty search personnel to identify and collect all possible terms and expressions related to the novelty search points by themselves, and then evaluate the retrieval results and adjust the retrieval strategy according to the retrieval results. Nor is it required that the retrieval personnel or novelty search personnel be proficient in various retrieval tools. Using the above-mentioned retrieval platform, relevant information can be located more quickly, greatly reducing the workload of the retrieval personnel or novelty search personnel and shortening the novelty search cycle, thereby improving the accuracy and efficiency of the retrieval.

[0139] Figure 3 It is a schematic flowchart of a retrieval method provided for an application embodiment of the present disclosure; as Figure 3 shown, the user inputs the information to be retrieved, and the LLM (Large Language Model, here, the large language model is a retrieval-based generation model) generates the first-round retrieval formula (referred to as the first retrieval formula) according to the information to be retrieved, and provides the first retrieval formula to the retrieval system; the retrieval system retrieves the first retrieval result according to the first retrieval formula and sends the first retrieval result to the evaluation system; the evaluation system evaluates the first retrieval result to obtain the first retrieval result evaluation (equivalent to the evaluation result); the first retrieval result evaluation is provided to the retrieval formula generation model, and the retrieval formula generation model updates the first retrieval formula according to the information to be retrieved, the first retrieval formula, the first retrieval result, the first retrieval result evaluation, and the relevant thesaurus, such as performing operations such as adding, deleting, and modifying the retrieval formula, to obtain the second retrieval formula;

[0140] The retrieval formula generation model provides the second retrieval formula to the retrieval system; the retrieval system retrieves the second retrieval result according to the second retrieval formula and sends the second retrieval result to the evaluation system; the evaluation system evaluates the retrieval result to obtain the second retrieval result evaluation; the second retrieval result evaluation is provided to the retrieval formula generation model, and the retrieval formula generation model updates the second retrieval formula according to the information to be retrieved, the second retrieval formula, the second retrieval result, the second retrieval result evaluation, and the relevant thesaurus, to obtain the third retrieval formula;

[0141] And so on. Subsequently, the retrieval formula generation model can update the i-th retrieval formula according to the information to be retrieved, the i-th retrieval formula, the i-th retrieval result, the i-th retrieval result evaluation, and the relevant thesaurus to obtain the (i + 1)-th retrieval formula for retrieval; until the retrieval formula generation model obtains a retrieval result that enables the retrieval system to obtain a corresponding evaluation result that meets the target condition, that is, the retrieval formula generation model continuously learns and updates, and the obtained retrieval formula enables the retrieval system to obtain a retrieval result, and the evaluation result meets the target condition after the evaluation system evaluates the retrieval result. That is, the retrieval formula finally generated by the retrieval formula generation model is the retrieval formula used to retrieve the answer output, and the retrieval result finally obtained by the retrieval system is the answer to the information to be retrieved.

[0142] Figure 4 A schematic flowchart of another retrieval method provided for an application embodiment of the present disclosure; as Figure 4 shown, the retrieval method includes:

[0143] Step 401, a user inputs information to be retrieved;

[0144] Step 402, a retrieval formula generation model generates a first retrieval formula for the first round, and provides the first retrieval formula to a retrieval system; the retrieval system retrieves a first retrieval result according to the first retrieval formula, and sends the first retrieval result to an evaluation system; the evaluation system evaluates the retrieval result to obtain a first retrieval result evaluation;

[0145] If the first retrieval result evaluation meets the target condition, go to step 405; if it does not meet the target condition, go to step 403;

[0146] Step 403, provide the i-th retrieval result evaluation to the retrieval formula generation model, and the retrieval formula generation model updates the i-th retrieval formula according to the information to be retrieved, the i-th retrieval formula, the i-th retrieval result, the i-th retrieval result evaluation, and a relevant thesaurus to obtain an (i + 1)-th retrieval formula, and provide the (i + 1)-th retrieval formula to the retrieval system;

[0147] Step 404, the retrieval system retrieves an (i + 1)-th retrieval result according to the (i + 1)-th retrieval formula, and sends the (i + 1)-th retrieval result to the evaluation system; the evaluation system evaluates the retrieval result to obtain an (i + 1)-th retrieval result evaluation;

[0148] If the (i + 1)-th retrieval result evaluation meets the target condition, go to step 405; if it does not meet the target condition, return to step 403, enter the next round of retrieval formula update, and re-update the retrieval formula according to the information to be retrieved, and the retrieval formula, retrieval result, retrieval result evaluation, and relevant thesaurus of this round.

[0149] For example, the i-th retrieval formula is: (intelligent cabinet AND intelligent storage cabinet) AND (dangerous goods OR dangerous substances OR dangerous chemicals);

[0150] Adjust the first retrieval item among them: add the keyword "intelligent storage cabinet" according to the relevant thesaurus;

[0151] After analysis, for the second retrieval item: "dangerous chemicals" leads to retrieving irrelevant documents and should be deleted; add a third retrieval item: NOT (dangerous chemicals AND dangerous reagents);

[0152] Then, the updated (i + 1)-th retrieval formula can be: (intelligent cabinet AND intelligent storage cabinet AND intelligent storage cabinet) AND (dangerous goods OR dangerous substances) NOT (dangerous chemicals AND dangerous reagents).

[0153] Step 405, present the search formula and search results.

[0154] Figure 5 It is a schematic structural diagram of the retrieval device provided by the embodiments of the present disclosure; as Figure 5 shown, the device includes:

[0155] An acquisition module, configured to acquire information to be retrieved;

[0156] A first processing module, configured to generate a first search formula according to the information to be retrieved;

[0157] The acquisition module is further configured to acquire a first search result and an evaluation result;

[0158] The first processing module is further configured to generate a second search formula according to the information to be retrieved, the first search result and the evaluation result; the first search result is retrieved by the search system according to the first search formula, the evaluation result is obtained by the evaluation system according to the first search result, and the second search formula is used to obtain the next round of search results by the search system.

[0159] In some embodiments, the first processing module is further configured to:

[0160] If the evaluation result does not meet the target condition, perform at least one round of search formula update operation until a search result whose evaluation result meets the target condition is obtained; each round of the search formula update operation is used to update the search formula by using the large language model; the updated search formula is used to re-perform the search by the search system.

[0161] In some embodiments, the acquisition module is further configured to determine the relevant thesaurus of each search term in the search formula used in the previous round;

[0162] The first processing module is configured to update the search formula according to the information to be retrieved, the search formula, the search result, the evaluation result used in the previous round, and the relevant thesaurus of each search term in the search formula.

[0163] It can be understood that when implementing the corresponding retrieval method, the retrieval device provided in the above embodiments can, according to needs, allocate the above processing to different program modules to complete all or part of the processing described above. In addition, the device provided in the above embodiments and the embodiments of the corresponding method belong to the same concept, and the specific implementation process is detailed in the method embodiments and will not be repeated here.

[0164] The embodiments of the present disclosure provide a computer-readable storage medium storing executable instructions, where the executable instructions, when executed by a processor, will trigger the processor to execute the retrieval method provided by the embodiments of the present disclosure.

[0165] In some embodiments, the computer-readable storage medium may be a ferroelectric random access memory (FRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or a memory such as a CD-ROM; it may also be various devices including one or any combination of the above memories.

[0166] In some embodiments, the executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as an independent program or being deployed as a module, model, subroutine, or other unit suitable for use in a computing environment.

[0167] As an example, the executable instructions may be deployed to execute on one computing device, or on multiple computing devices located at one location, or alternatively, on multiple computing devices distributed at multiple locations and interconnected by a communication network.

[0168] The embodiments of the present disclosure provide a computer program product, the computer program product includes a computer program / instructions, and when the computer program / instructions are executed by a processor, the retrieval method described in the present disclosure is implemented.

[0169] Figure 6 It is a schematic structural diagram of the retrieval platform provided by the embodiments of the present disclosure; as Figure 6 shown, the retrieval platform includes: a large language model, a retrieval system, and an evaluation system;

[0170] The large language model is used to obtain the information to be retrieved and determine a first retrieval formula according to the information to be retrieved;

[0171] The retrieval system is used to perform a retrieval according to the first retrieval formula and obtain a first retrieval result;

[0172] The evaluation system is used to evaluate the first retrieval result and obtain an evaluation result.

[0173] In some embodiments, the large language model is further configured to, if the evaluation result does not meet the target condition, perform at least one round of retrieval-based update operations until a retrieval result whose evaluation result meets the target condition is obtained; each round of the retrieval-based update operation is used to update the retrieval formula by using the large language model; the updated retrieval formula is used for the retrieval system to perform retrieval again.

[0174] In some embodiments, the evaluation system is configured to perform at least one of the following:

[0175] Identify the literature metadata of each retrieved document in the information to be retrieved and the first retrieval result by using a first model, and obtain the relevance score of each retrieved document; determine the number of documents whose relevance score exceeds a threshold and / or the average relevance score according to the relevance score of each retrieved document;

[0176] Identify the document content of each retrieved document in the information to be retrieved and the first retrieval result by using a second model, and obtain the content relevance score of each retrieved document; determine the retrieval quality score according to the content relevance score of each retrieved document;

[0177] Receive the adoption feedback from the user for the first retrieval result, and determine the proportion of accepted documents according to the adoption feedback.

[0178] In some embodiments, the evaluation result meeting the target condition includes at least one of the following:

[0179] The number of documents whose relevance score exceeds the threshold exceeds a first threshold;

[0180] The average relevance score exceeds a second threshold;

[0181] The retrieval quality score exceeds a third threshold;

[0182] The proportion of accepted documents exceeds a fourth threshold.

[0183] In some embodiments, the evaluation system is further configured to perform weighted calculation according to the number of documents whose relevance score exceeds the threshold, the average relevance score, the retrieval quality score, and / or the proportion of accepted documents, and obtain a weighted calculation result;

[0184] The evaluation result meeting the target condition further includes: the weighted calculation result exceeds a fifth threshold.

[0185] In some embodiments, the evaluation system further includes: an interaction system, configured to perform one of the following:

[0186] Present at least one round of retrieval formulas and the retrieval results corresponding to each round of the retrieval formulas, and select a target retrieval formula and a target retrieval result according to the user feedback;

[0187] Present a retrieval log, where the retrieval log includes: at least one round of retrieval expressions and the retrieval results corresponding to each round of retrieval expressions.

[0188] It can be understood that when implementing the corresponding retrieval method, the retrieval platform provided in the above embodiments can, as needed, allocate the above processing to different systems to complete all or part of the processing described above. In addition, the retrieval platform provided in the above embodiments and the embodiments of the corresponding method belong to the same concept. For the specific implementation process, please refer to the method embodiments, which will not be elaborated here.

[0189] Figure 7 Schematic structural diagram of an electronic device provided by an embodiment of the present disclosure; as Figure 7 shown, the electronic device 70 includes: a processor 701 and a memory 702 for storing a computer program that can run on the processor; when the processor 701 runs the computer program, it executes the retrieval method provided by the embodiment of the present disclosure.

[0190] In practical applications, the electronic device 70 may further include: at least one network interface 703. Each component in the electronic device 70 is coupled together through a bus system 704. It can be understood that the bus system 704 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 704 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in Figure 7 all kinds of buses are labeled as the bus system 704. Among them, the number of the processors 701 can be at least one. The network interface 703 is used for wired or wireless communication between the electronic device 70 and other devices.

[0191] The memory 702 in the embodiment of the present disclosure is used to store various types of data to support the operation of the electronic device 70.

[0192] The methods disclosed in the above embodiments of the present disclosure can be applied to or implemented by the processor 701. The processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the above methods can be completed by the integrated logic circuit in the hardware of the processor 701 or instructions in the form of software. The above-mentioned processor 701 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 701 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present disclosure. The general-purpose processor may be a microprocessor or any conventional processor, etc. Combining the steps of the methods disclosed in the embodiments of the present disclosure, it can be directly embodied as being executed by the hardware decoding processor, or completed by a combination of the hardware and software modules in the decoding processor. The software module may be located in the storage medium, and this storage medium is located in the memory 702. The processor 701 reads the information in the memory 702 and combines its hardware to complete the steps of the foregoing methods.

[0193] In some embodiments, the electronic device 70 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontroller units (MCUs), microprocessors, or other electronic components for executing the foregoing methods.

[0194] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and no limitations are imposed herein.

[0195] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of the present disclosure, "a plurality" means two or more unless otherwise specifically defined.

[0196] As described above, this is only a specific implementation manner of the present disclosure. However, the protection scope of the present disclosure is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present disclosure can easily think of changes or substitutions, which should all be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims described above.

Claims

1. A retrieval method, applied to a large language model, comprising: Get the information to be retrieved; Generate a first search formula according to the information to be searched; Obtaining a first search result and an evaluation result, and generating a second search formula according to the information to be searched, the first search result and the evaluation result; The first search result is obtained by the search system according to the first search formula, the evaluation result is obtained by the evaluation system according to the first search result, and the second search formula is used by the search system to obtain the next round of search results.

2. The method according to claim 1, further comprising: If the evaluation result does not meet the target condition, perform at least one round of search-type update operation until a search result is obtained in which the evaluation result meets the target condition; Each round of the search formula updating operation is used to update the search formula using the large language model; the updated search formula is used for re-searching by the search system.

3. The method according to claim 2, further comprising: Determine the relevant thesaurus for each search term in the search formula used in the previous round; Perform at least one round of search-based update operations, including: The search formula is updated according to the information to be searched, the search formula used in the previous round, the search results, the evaluation results and the related thesaurus of each search term in the search formula.

4. A search method, the method being applied to a search platform, the search platform comprising: Large language model, retrieval system, evaluation system; the method comprises: The large language model obtains information to be searched, and determines a first search formula according to the information to be searched; The retrieval system performs a search according to the first search formula to obtain a first search result; The evaluation system evaluates the first search result to obtain an evaluation result.

5. The method according to claim 4, further comprising: If the evaluation result does not meet the target condition, perform at least one round of search-type update operation until a search result is obtained in which the evaluation result meets the target condition; Each round of the search formula updating operation is used to update the search formula using the large language model; the updated search formula is used for re-searching by the search system.

6. The method according to claim 4, wherein evaluating the first search result comprises at least one of the following: Using the first model to identify the information to be retrieved and the document metadata of each retrieved document in the first search result, and obtaining a relevance score of each retrieved document; according to the relevance score of each retrieved document, determining the number of documents whose relevance score exceeds a threshold and / or an average relevance score; Using the second model to identify the information to be retrieved and the document content of each retrieved document in the first search result, and obtaining a content relevance score for each retrieved document; Determine the search quality score based on the content relevance score of each searched document; Receive user adoption feedback on the first search result, and determine the proportion of accepted documents based on the adoption feedback.

7. According to the method of claim 5, the evaluation result satisfies the target condition, including at least one of the following: The number of documents with relevance scores exceeding the threshold, exceeding the first threshold; The average relevance score,exceeds the second threshold; The search quality score exceeds the third threshold; The proportion of accepted documents exceeds the fourth threshold.

8. The method according to claim 7, evaluating the first search result, further comprising: Performing weighted calculation according to the number of documents with relevance scores exceeding a threshold, the average relevance score, the search quality score and / or the proportion of the accepted documents to obtain a weighted calculation result; The evaluation result satisfies the target condition and further includes: the weighted calculation result exceeds the fifth threshold.

9. The method according to claim 4, further comprising one of the following: Present at least one round of search formulas and search results corresponding to each round of the search formulas, and select a target search formula and target search results based on user feedback; A search log is presented, the search log comprising: At least one round of search formulas and search results corresponding to each round of search formulas.

10. A retrieval device, comprising: An acquisition module, used to acquire information to be retrieved; A first processing module, used for generating a first search formula according to the information to be searched; The acquisition module is further used to acquire the first search result and the evaluation result; The first processing module is also used to generate a second search formula based on the information to be retrieved, the first search result and the evaluation result; the first search result is obtained by the retrieval system according to the first search formula, the evaluation result is obtained by the evaluation system according to the first search result, and the second search formula is used by the retrieval system to obtain the next round of search results.

Images