Search strategy evaluation method, device, and storage medium

Abstract

The application provides a retrieval strategy evaluation method, device and storage medium, which can be applied to the technical field of retrieval. The method comprises the following steps: in response to receiving a retrieval strategy update instruction, combining a plurality of retrieval rules matched with retrieval configuration parameters to obtain a plurality of retrieval strategies, the retrieval configuration parameters indicating the retrieval type of the plurality of retrieval strategies and the retrieval condition matched with the retrieval type; according to a sample question, performing retrieval in a preset knowledge base by using the plurality of retrieval strategies to obtain a plurality of target authentication information; processing the plurality of target authentication information based on the sample question by using a preset large model to output a plurality of replies to the sample question; based on the plurality of replies, evaluating the retrieval strategy corresponding to each of the plurality of target authentication information to obtain a plurality of evaluation results of the plurality of retrieval strategies; based on the plurality of evaluation results, screening in the plurality of retrieval strategies to obtain a target retrieval strategy, and taking the target retrieval strategy as an updated retrieval strategy.

Description

Technical Field

[0001] This application relates to the field of retrieval technology, specifically to retrieval strategy testing in authentication scenarios, and more specifically to a retrieval strategy evaluation method, device, and storage medium. Background Technology

[0002] The authentication scenario involves a large number of documents, many versions, and frequent updates. Retrieval-augmented generation (RAG) technology can be applied to the authentication domain. However, this technology is updated and iterated rapidly and has a wide variety of types. Fixed retrieval strategies have limited accuracy in document retrieval in the authentication domain. To improve retrieval accuracy, retrieval strategies need to be adjusted.

[0003] In the process of adjusting and testing search strategies, traditional methods often involve enriching the search strategy by manually selecting or updating the database. The former relies on subjective human experience, while the latter is limited by the database, making it difficult to achieve rapid combination and effective verification of search methods. Summary of the Invention

[0004] In view of the above problems, this application provides a method, apparatus, device and storage medium for improving the efficiency of search strategy generation and evaluation.

[0005] According to a first aspect of this application, a retrieval strategy evaluation method is provided. The method includes: in response to receiving a retrieval strategy update instruction, combining multiple retrieval rules matching retrieval configuration parameters to obtain multiple retrieval strategies, wherein the retrieval configuration parameters indicate the retrieval type of the multiple retrieval strategies and the retrieval conditions matching the retrieval type; based on a sample question, using the multiple retrieval strategies to search in a preset knowledge base to obtain multiple target authentication information, wherein the target authentication information indicates authentication domain knowledge matching the sample question; processing the multiple target authentication information based on the sample question using a preset large model to output multiple responses to the sample question; evaluating the retrieval strategies corresponding to each of the multiple target authentication information based on the multiple responses to obtain multiple evaluation results for the multiple retrieval strategies; filtering among the multiple retrieval strategies based on the multiple evaluation results to obtain a target retrieval strategy, and using the target retrieval strategy as the updated retrieval strategy to search in the preset knowledge base in response to receiving a question to obtain authentication domain knowledge matching the question.

[0006] According to embodiments of this application, combining multiple search rules matching search configuration parameters to obtain multiple search strategies includes: filtering multiple preset candidate search rule templates based on the search type indicated by the search configuration parameters to obtain a target search rule template, wherein the target search rule template indicates the hierarchical data structure corresponding to the multiple search rules; generating multiple candidate values ​​for the search configuration parameters based on the numerical type corresponding to the search configuration parameters and at least one of the parameter value ranges matching the numerical type, wherein the numerical type includes at least one of integer, floating-point number, boolean value, string, range parameter, and no range parameter; combining the multiple candidate values ​​to obtain multiple candidate value sets; mapping the multiple candidate value sets to search rule templates respectively to obtain multiple search rules; and combining the multiple search rules to obtain multiple search strategies.

[0007] According to an embodiment of this application, generating multiple candidate values ​​for the retrieval configuration parameters based on the parameter type and value range corresponding to the retrieval configuration parameters includes: filtering from multiple preset candidate value rules based on the numerical type to obtain a target value rule, wherein each of the multiple candidate value rules has a matching numerical type; and generating multiple candidate values ​​for the retrieval configuration parameters within the parameter value range using the target value rule, wherein the target value rule is a full traversal rule, a switch-type value rule, an integer step traversal rule, a floating-point step traversal rule, a single selection rule, a multiple selection rule, or a no-selection rule.

[0008] According to embodiments of this application, based on a sample question, multiple retrieval strategies are used to search a preset knowledge base to obtain multiple target authentication information, including: obtaining question processing prompts based on question processing configuration parameters; guiding a preset large model to process the sample question based on question processing rules indicated by the question processing prompts to obtain multiple sub-sample questions, wherein the question processing rules include at least one of question rewriting, question association, and language type conversion; converting the multiple sub-sample questions into multiple sets of retrieval questions based on data representation forms indicated by multiple retrieval strategies, wherein the data representation forms include at least one of keyword sets, sparse vectors, dense vectors, structured metadata, and knowledge graphs; and using multiple retrieval strategies to search the preset knowledge base to obtain target authentication information matching the multiple sets of retrieval questions, wherein the retrieval types of the multiple retrieval strategies include at least one of dense retrieval, sparse retrieval, hybrid retrieval, keyword retrieval, filter retrieval, and knowledge graph retrieval.

[0009] According to embodiments of this application, a pre-defined large model is used to process multiple target authentication information based on a sample question, and multiple responses to the sample question are output. This includes: filtering multiple pre-defined candidate prompt word templates based on the evaluation stages corresponding to the multiple target authentication information to obtain target prompt word templates, each of the multiple candidate prompt word templates having a matching evaluation stage, the evaluation stage indicating the evaluation stage of the multiple target authentication information; mapping the sample question, historical conversation information, and multiple target authentication information to the target prompt word templates respectively to obtain multiple question-and-answer prompt words; and processing the multiple question-and-answer prompt words using the pre-defined large model to obtain multiple responses to the sample question.

[0010] According to embodiments of this application, based on multiple responses, the retrieval strategies corresponding to each of the multiple target authentication information are evaluated to obtain multiple evaluation results for the multiple retrieval strategies, including: filtering multiple preset candidate evaluation prompts based on the retrieval types indicated by the multiple retrieval strategies to obtain multiple target evaluation prompts, each of the multiple candidate evaluation prompts having a matching retrieval type; and guiding a preset large model to process multiple target authentication information, multiple responses, and a preset standard answer based on the evaluation rules indicated by each of the multiple target evaluation prompts to obtain multiple evaluation results for the retrieval strategies.

[0011] According to embodiments of this application, based on multiple responses, the retrieval strategies corresponding to each of the multiple target authentication information are evaluated to obtain multiple evaluation results for the multiple retrieval strategies, including: obtaining multiple process evaluation scores based on the degree of matching between the multiple target authentication information and the preset standard answer, wherein the process evaluation scores indicate the retrieval effectiveness of the corresponding retrieval strategy; obtaining multiple result evaluation scores based on the degree of matching between the multiple target authentication information and the preset standard answer and the degree of matching between the multiple responses and the preset standard answer, wherein the result evaluation scores indicate the degree of use of the corresponding target authentication information when generating the response; and fusing the multiple process evaluation scores and the multiple answer evaluation scores to obtain multiple evaluation results for the multiple retrieval strategies.

[0012] According to an embodiment of this application, a preset knowledge base includes multiple candidate authentication information. The preset knowledge base is obtained by: obtaining document processing prompt words based on document processing configuration parameter-indicated document processing prompt word templates; guiding a preset large model to process multiple authenticated documents based on document processing prompt words-indicated document processing rules, obtaining multiple metadata of multiple authenticated documents, and associating the multiple metadata with the corresponding authenticated documents. The document processing rules include at least one of extracting document keywords and summarizing document content, and the metadata includes at least one of summaries and keywords; dividing the text data in the multiple authenticated documents into multiple text blocks, and vectorizing the multiple text blocks through a pre-trained embedding model, obtaining multiple document vectors as candidate authentication information to form the preset knowledge base.

[0013] A second aspect of this application provides a retrieval strategy evaluation apparatus, comprising: an acquisition module, configured to, in response to receiving a retrieval strategy update instruction, combine multiple retrieval rules matching retrieval configuration parameters to obtain multiple retrieval strategies, wherein the retrieval configuration parameters indicate the retrieval types of the multiple retrieval strategies and retrieval conditions matching the retrieval types; a retrieval module, configured to, based on a sample question, perform a retrieval in a preset knowledge base using the multiple retrieval strategies to obtain multiple target authentication information, wherein the target authentication information indicates authentication domain knowledge matching the sample question; a question-answering module, configured to, using a preset large model, process the multiple target authentication information based on the sample question and output multiple responses to the sample question; an evaluation module, configured to, based on the multiple responses, evaluate the retrieval strategies corresponding to each of the multiple target authentication information to obtain multiple evaluation results for the multiple retrieval strategies; and a filtering module, configured to, based on the multiple evaluation results, filter among the multiple retrieval strategies to obtain a target retrieval strategy, and use the target retrieval strategy as the updated retrieval strategy to, in response to receiving a question, perform a retrieval in the preset knowledge base using the target retrieval strategy to obtain authentication domain knowledge matching the question.

[0014] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0015] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0016] According to the retrieval strategy evaluation method, apparatus, device, and storage medium provided in the embodiments of this application, upon receiving a retrieval strategy update instruction, multiple retrieval rules are combined according to the retrieval type indicated by the retrieval configuration parameters and the retrieval conditions matching the retrieval type to obtain multiple retrieval strategies. This enables rapid combination of retrieval strategies and improves the diversity of retrieval strategies. Multiple target authentication information is retrieved using multiple retrieval strategies, providing authentication domain knowledge for the sample problem to the preset large model. This allows the preset large model to effectively evaluate the retrieval strategies based on the target authentication information and responses within the scope of authentication domain knowledge, realizing automated and rapid testing from retrieval strategy combination to evaluation. The obtained target retrieval strategies are used as updated retrieval strategies to achieve retrieval strategy switching, thereby meeting different user retrieval needs. Attached Figure Description

[0017] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0018] Figure 1 The illustration shows an application scenario diagram of the retrieval strategy evaluation method, apparatus, device, medium, and program product according to embodiments of this application;

[0019] Figure 2 A flowchart illustrating a retrieval strategy evaluation method according to an embodiment of this application is shown schematically.

[0020] Figure 3 A flowchart illustrating the process of obtaining multiple retrieval strategies according to embodiments of this application is shown in the illustration.

[0021] Figure 4 A flowchart illustrating the generation of multiple candidate values ​​according to an embodiment of this application is shown schematically;

[0022] Figure 5 A detailed flowchart illustrating a retrieval strategy evaluation method according to an embodiment of this application is shown schematically.

[0023] Figure 6 A detailed flowchart illustrating the construction of a preset knowledge base according to an embodiment of this application is shown schematically;

[0024] Figure 7 A flowchart illustrating sample problem processing according to an embodiment of this application is shown schematically;

[0025] Figure 8 A detailed flowchart illustrating the execution of a retrieval strategy according to an embodiment of this application is shown schematically;

[0026] Figure 9 A detailed flowchart illustrating a model question-and-answer format according to an embodiment of this application is shown schematically.

[0027] Figure 10 This schematically illustrates a structural block diagram of a retrieval strategy evaluation apparatus according to an embodiment of this application; and,

[0028] Figure 11 A block diagram schematically illustrates an electronic device suitable for implementing a retrieval strategy evaluation method according to an embodiment of this application. Detailed Implementation

[0029] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0030] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0031] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0032] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0033] In the large-scale question-answering process in the certification field, there are a large number of certification documents, many language versions and year versions, and certification documents are constantly updated over time, with changes in the overall document characteristics. RAG has a fast update and iteration speed and a wide variety of types, which can be applied to the retrieval of certification documents. However, relying on its fixed retrieval strategy makes it difficult to achieve correct retrieval from the complex, diverse and changing documents in the certification field, which leads to the need for constant adjustment of the retrieval strategy.

[0034] Related technologies adjust or update search strategy parameters or integrate search strategies by manually selecting or updating the database. The former relies on human subjective experience and has high time and manpower costs, while the latter is limited by the database and makes it difficult to freely combine and verify search methods, which has great limitations in determining the optimal search strategy.

[0035] Furthermore, the RAG retrieval frameworks or RAG evaluation frameworks in related technologies do not have the function of automated retrieval and evaluation.

[0036] To address at least one of the aforementioned technical problems, embodiments of this application provide a retrieval strategy evaluation method. This method utilizes received retrieval configuration parameters to rapidly combine retrieval rules, providing users with multiple retrieval strategies to meet their needs. By employing these multiple retrieval strategies, target authentication information containing authentication domain knowledge is retrieved from a preset knowledge base. This provides a scope of authentication domain knowledge for answering questions and evaluating retrieval strategies in a preset large model, thereby achieving automated and rapid testing of retrieval strategy combination, application, and evaluation in the authentication domain.

[0037] Figure 1 The diagram illustrates an application scenario of the retrieval strategy evaluation method according to an embodiment of this application.

[0038] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0039] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0040] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0041] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0042] It should be noted that the retrieval strategy evaluation method provided in this application embodiment can generally be executed by server 105. Correspondingly, the retrieval strategy evaluation device provided in this application embodiment can generally be located in server 105. The retrieval strategy evaluation method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the retrieval strategy evaluation device provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0043] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0044] The following will be based on Figure 1 The described scene, through Figures 2-9 The retrieval strategy evaluation method according to the embodiments of this application will be described in detail.

[0045] Figure 2 A flowchart illustrating a retrieval strategy evaluation method according to an embodiment of this application is shown.

[0046] like Figure 2 As shown, the retrieval strategy evaluation method of this embodiment includes operations S210 to S250.

[0047] In operation S210, in response to receiving a search strategy update instruction, multiple search rules that match the search configuration parameters are combined to obtain multiple search strategies.

[0048] According to embodiments of this application, the retrieval strategy update instruction indicates that the user needs to update the retrieval strategy applied in the large-scale question-answering process due to changes in retrieval needs, application scenarios, etc., and may refer to a trigger instruction generated in response to a user's confirmation operation. The confirmation operation may indicate the user's click on the update button.

[0049] The search configuration parameters indicate the search type for multiple search strategies and the search criteria that match the search type. The search type can be the type of search engine included in the search strategy, such as a hybrid search engine, dense vector search engine, keyword search engine, filter search engine, and knowledge graph search engine. The search criteria that match the search type can be detailed configuration parameters of the search engine corresponding to the search engine type, such as the number of documents retrieved, whether to enable the re-ranking model, and the re-ranking model score threshold.

[0050] Retrieval rules can be predefined execution logic and steps of a retrieval tool that matches the corresponding retrieval tool type. For example, if the retrieval tool type is a dense vector retrieval tool, the retrieval rules can include receiving questions, vectorizing the questions, calculating the similarity score between the obtained question vector and the candidate document vectors in the predefined knowledge base, and sorting the search results in descending order of similarity score. Retrieval strategies can be retrieval execution schemes composed of retrieval rules.

[0051] For example, based on the numeric type corresponding to the retrieval configuration parameters, candidate values ​​are generated and combined within the parameter value range that matches the numeric type. The resulting combinations of multiple candidate values ​​are mapped to retrieval rule templates that match the retrieval type, resulting in multiple retrieval rules. Random combinations of one or more of these retrieval rules are used as multiple retrieval strategies.

[0052] For example, multiple candidate search rules can be filtered based on search configuration parameters and predefined mapping relationships to obtain multiple search rules that match the search configuration parameters.

[0053] According to embodiments of this application, the mapping relationship indicates the correspondence between retrieval configuration parameters and retrieval rules. For example, the retrieval configuration parameters can be a dense vector retrieval tool and its detailed configuration parameters, while the retrieval rules can be the execution logic and steps of the dense vector retrieval tool.

[0054] In operation S220, based on the sample question, multiple retrieval strategies are used to search in the preset knowledge base to obtain multiple target authentication information.

[0055] According to embodiments of this application, the sample question can be a question input by testers, and the same sample question is used when testing different retrieval strategies. The preset knowledge base can be a database storing a massive number of certification documents of different versions and years, used to provide certification domain knowledge for RAG technology, ensuring that the final optimal retrieval strategy has good retrieval performance in the certification domain application. The target certification information indicates the certification domain knowledge matching the sample question, and may include multiple certification documents retrieved that match the sample question. Certification refers to conformity assessment activities that prove that products, services, and management systems comply with relevant standards and technical specifications, and may include product certification, management system certification, and service certification.

[0056] For example, the sample question could be an authentication information query, etc. Authentication documents can be stored as document vectors in a pre-defined knowledge base. The target authentication information can be relevant paragraphs from the authentication document, represented as vectors.

[0057] In operation S230, a pre-set large model is used to process multiple target authentication information based on sample questions, and output multiple responses to the sample questions.

[0058] According to embodiments of this application, a pre-defined large model can refer to a deep learning model with a large number of model parameters. A large model typically contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters.

[0059] For example, the large model may include a large language model (LLM), a multimodal large model, etc. The large model involved in the embodiments of this application may be a general large model, or it may be an expert large model obtained by fine-tuning based on requirements. The embodiments of this application do not limit this.

[0060] In operation S240, based on multiple responses, the retrieval strategies corresponding to each of the multiple target authentication information are evaluated, resulting in multiple evaluation results for the multiple retrieval strategies.

[0061] According to embodiments of this application, multiple evaluation results are also the test results of their respective retrieval strategies, characterizing the retrieval performance of their respective retrieval strategies.

[0062] For example, the evaluation result can be an evaluation score. The semantic analysis capability of the pre-set large model is used to evaluate the corresponding retrieval strategies based on multiple target authentication information and multiple responses, resulting in multiple evaluation scores for multiple retrieval strategies.

[0063] In operation S250, based on multiple evaluation results, multiple retrieval strategies are selected to obtain the target retrieval strategy. The target retrieval strategy is used as the updated retrieval strategy in response to the received question. The target retrieval strategy is used to search the preset knowledge base to obtain the authentication domain knowledge that matches the question.

[0064] For example, multiple evaluation scores are sorted in descending order, and the retrieval strategy corresponding to the highest evaluation score is used as the target retrieval strategy, which is the updated retrieval strategy.

[0065] Through operations S210-S250, upon receiving a retrieval strategy update instruction, the system can quickly invoke matching retrieval rules based on the input retrieval configuration parameters, generating multiple retrieval strategies to be tested, thus improving the speed of retrieval strategy generation and combination. By executing multiple retrieval strategies, multiple target authentication information is obtained, providing the authentication domain knowledge scope for the large model to answer sample questions. The large model's response and target authentication information are then used to evaluate each retrieval strategy, achieving one-stop automated testing from retrieval strategy self-combination and application to evaluation. The obtained target retrieval strategy is used as the updated retrieval strategy, enabling retrieval strategy switching and meeting different user retrieval needs.

[0066] Figure 3 A flowchart illustrating the process of obtaining multiple retrieval strategies according to an embodiment of this application is shown.

[0067] The following will combine Figure 3 This describes the process of generating multiple search strategies. In some embodiments, multiple search rules 303 that match the search configuration parameters are combined to obtain multiple search strategies 304, including:

[0068] First, based on the search type indicated by the search configuration parameters, multiple preset candidate search rule templates 301 are filtered to obtain the target search rule template 302.

[0069] According to an embodiment of this application, each of the multiple candidate retrieval rule templates 301 has a matching retrieval type, and the candidate retrieval rule template indicates the hierarchical data structure corresponding to the multiple retrieval rules.

[0070] For example, multiple candidate search rule templates 301 can be encapsulated into a JSON-formatted configuration file. The first layer of the hierarchical data structure can be the searcher type matched by the corresponding search rule and its position in the configuration file; the second layer can be the search conditions corresponding to the searcher and their position in the configuration file.

[0071] Taking a hybrid search engine as an example, in the hybrid search engine configuration file, the first layer can be the search type, that is, the search engine type, and its configuration value can be "hybrid search engine". The second layer can be the search conditions corresponding to the hybrid search engine, that is, the detailed configuration parameters of the hybrid search engine, which can include the configuration parameters of the dense search engine, the configuration parameters of the sparse search engine, the number of search recalls, etc.

[0072] Then, based on at least one of the numerical type 305 corresponding to the retrieval configuration parameter and the parameter value range 308 matching the numerical type, multiple candidate values ​​306 for the retrieval configuration parameter are generated.

[0073] According to embodiments of this application, the numeric type includes at least one of integer, floating-point number, Boolean value, string, range parameter, and no range parameter.

[0074] For example, within the value range of the numeric type corresponding to each search configuration parameter, multiple candidate values ​​306 are randomly generated for each search configuration parameter.

[0075] Then, the multiple candidate values ​​are combined to obtain a set of multiple candidate values ​​307.

[0076] Then, the multiple candidate value sets 307 are mapped to the retrieval rule templates respectively to obtain multiple retrieval rules 303.

[0077] For example, multiple candidate values ​​are combined to obtain multiple candidate value sets 307. The candidate values ​​in each candidate value set are mapped to the corresponding positions in the retrieval rule template according to the retrieval type parameters corresponding to the candidate values, to obtain multiple retrieval rules 303, such as retrieval rule_1 to retrieval rule_n.

[0078] Then, multiple search rules 303 are combined to obtain multiple search strategies 304.

[0079] For example, search rules can be randomly combined in a single-choice, multiple-choice, or other combination manner to obtain multiple search strategies.

[0080] In the embodiments of this application, based on the retrieval configuration parameters that reflect the testing needs of testers, a pre-configured retrieval rule template is called to automatically combine and generate multiple retrieval strategies. This improves the generation speed of retrieval strategies while increasing the diversity of retrieval strategies, thus providing rich samples for retrieval strategy testing.

[0081] Figure 4 A flowchart illustrating the generation of multiple candidate values ​​according to an embodiment of this application is shown.

[0082] The following will combine Figure 4This describes the process for generating candidate values. In some embodiments, generating multiple candidate values ​​for the retrieval configuration parameter based on at least one of the numeric type corresponding to the retrieval configuration parameter and the parameter value range matching the numeric type includes:

[0083] First, based on the numerical type, the target numerical generation rule 402 is obtained by filtering from multiple preset candidate numerical generation rules 401.

[0084] According to embodiments of this application, multiple candidate value generation rules 401 each have a matching value type. The multiple candidate value generation rules 401 may include full traversal rules, switch-type value selection rules, integer step size traversal rules, floating-point step size traversal rules, single selection rules, multiple selection rules, and no selection rules, etc.

[0085] The search configuration parameters include search type parameters and search execution parameters that match the search type. The search type parameters indicate the search type, and the search execution parameters indicate the search conditions.

[0086] For example, a full traversal rule matches numeric types with no range parameters, and can traverse all search parameters. A switch-type value rule matches boolean values ​​(bool), where the corresponding parameter can be True or False, indicating enabled or disabled. An integer step traversal rule matches integer values ​​(int), where the value is selected between the maximum and minimum values ​​according to a preset step size. A floating-point step traversal rule matches floating-point numbers (float), where the value is selected between the maximum and minimum values ​​according to a preset step size. A single-selection rule matches scope parameters, where one parameter can be selected from a list of preset scope parameters. A multi-selection rule matches scope parameters, where multiple parameters can be selected from a list of corresponding scope parameters. A no-selection rule matches scope parameters, where no parameter is selected from a list of corresponding scope parameters.

[0087] Furthermore, each search configuration parameter is matched with an automated control parameter (is_optimize). The automated control parameter controls whether the corresponding parameter participates in the above parameter combination and value. The automated combination and value of the search configuration parameter will only take effect when is_optimize is 1.

[0088] Then, using the target value generation rule 402, multiple candidate values ​​306 for the retrieval configuration parameters are generated within the value range.

[0089] According to an embodiment of this application, the target value generation rule 402 is a full traversal rule, a switch-type value retrieval rule, an integer step size traversal rule, a floating-point step size traversal rule, a single selection rule, a multiple selection rule, or a no-selection rule.

[0090] Furthermore, when there are multiple numerical types corresponding to the retrieval configuration parameters or multiple candidate value rules corresponding to the numerical types, the number of target numerical generation rules 402 can be multiple, and the multiple target numerical generation rules 402 are different.

[0091] Furthermore, when the numeric type corresponding to the retrieved configuration parameter is a string, the corresponding string is directly used as the candidate value.

[0092] For example, when the retrieval configuration parameter is a retrieval type, the numerical type is a scope parameter, and the target numerical generation rule 402 can be a single-selection rule, a multiple-selection rule, and / or a no-selection rule. In a single-selection rule, one can be selected from the scope parameter list "Dense Vector Retrieval, Sparse Vector Retrieval, and Hybrid Retrieval". In a multiple-selection rule, multiple can be selected from "Dense Vector Retrieval, Sparse Vector Retrieval, and Hybrid Retrieval". In a no-selection rule, no selection is made in the scope parameter list; it is manually set by the tester. When the retrieval configuration parameter is a re-ranking model score threshold, the numerical type is a floating-point number. In the floating-point step size traversal rule, the maximum value can be 0.8, the minimum value can be 0.1, and the preset step size can be 0.1.

[0093] Through the above operations, the values ​​of the retrieval configuration parameters are combined using predefined numerical generation rules that match their respective numerical types. This enables the rapid assignment and combination of each parameter, facilitating the direct execution of different retrieval strategies and the rapid verification of their recall results in subsequent searches. This greatly improves the efficiency of selecting the optimal retrieval strategy during the testing phase.

[0094] Figure 5 A detailed flowchart illustrating a retrieval strategy evaluation method according to an embodiment of this application is shown in the schematic diagram. Figure 6 A detailed flowchart illustrating the construction of a preset knowledge base according to an embodiment of this application is shown schematically.

[0095] To test retrieval strategies using massive amounts of authentication domain knowledge, ensure the accuracy of retrieval within the authentication domain, and improve the efficiency of building the pre-defined knowledge base, the following will combine... Figure 5 , Figure 6 This explains the construction process of the pre-defined knowledge base 503.

[0096] In some embodiments, the preset knowledge base 503 includes multiple candidate authentication information, which is obtained by the preset knowledge base 503 in the following manner:

[0097] First, document processing prompts are obtained based on the document processing prompt template indicated by the document processing configuration parameters.

[0098] According to an embodiment of this application, the document processing configuration parameters indicate the execution status of the document processing prompt word templates, that is, whether the document processing rules corresponding to each document processing prompt word template are executed when processing authentication documents.

[0099] For example, document processing configuration parameters can be set by testers according to testing needs, using the document processing prompt word template corresponding to the document processing rule indicated in the document processing configuration parameters as the document processing word. The document processing prompt word template can also be encapsulated as a JSON format configuration file, and the corresponding configuration file can be directly called in response to the document processing prompt word template indicated by the document processing configuration parameters, thereby improving the efficiency of document processing prompt word generation.

[0100] For example, document processing configuration parameters can also be generated based on their corresponding numeric types and the value selection rules that match the numeric types. For instance, if the numeric type of the document processing configuration parameter is the scope parameter, then a single-choice rule or a multi-choice rule can be used to select from the preset scope parameter list of "extract document keywords, summarize document content, or do no additional processing".

[0101] Based on this, document processing can be performed by directly calling the encapsulated document processing prompt templates that reflect the testers' testing needs using document processing configuration parameters, thus improving document processing efficiency while ensuring the effectiveness of document processing.

[0102] Then, based on the document processing rules indicated by the document processing prompt words, the preset large model 505 is guided to process the multiple certified documents 601 respectively, obtain multiple metadata of the multiple certified documents 601, and associate the multiple metadata with the corresponding certified documents respectively.

[0103] According to embodiments of this application, the document processing rules include at least one of extracting document keywords and summarizing document content, and the metadata includes at least one of the summary and keywords, which can serve as the polishing result of the certification document and reflect the core content of the certification document.

[0104] For example, document processing rules may include extracting document keywords, summarizing document content, and extracting keywords after summarizing document content, or simply extracting keywords, summarizing, and extracting keywords after summarizing. For example, the prompt for summarizing document content is "Summarize the document topic in one sentence (maximum 15 words), present its content for search purposes, and the summary must be consistent with the document language; context: {{documents.content}} Output format: summary:", where documents refers to the list of all certified documents, and content refers to the text content of each certified document.

[0105] Guided by the document processing prompts, the pre-defined large model 505 iteratively executes the aforementioned document processing rules on each certified document, obtaining multiple responses containing keywords and summaries, which are then stored in the metadata corresponding to each certified document as polishing results.

[0106] For example, before processing the authentication documents, multiple authentication documents 601 are serialized by a loader 602 of the corresponding document type, and a list of documents, namely the authentication document list 603, is output. When processing the authentication documents, the input is a list of documents and the output is a list of strings.

[0107] Then, the text data in multiple certified documents 601 is divided into multiple text blocks, and the multiple text blocks are vectorized by a pre-trained embedding model. The resulting document vectors are written into the vector library as candidate certification information to form a preset knowledge base 503.

[0108] For example, the text data includes the text content and metadata of the authentication document. The text data is segmented by a segmenter 604, and the segmentation method may include segmentation by page, segmentation by sentence, and segmentation by word count, etc.; the pre-trained embedding model can be bge-m3 (a general vector model); the document vector V(content) can be represented as:

[0109] V(content)=doc_embedder(content);

[0110] In the formula, doc_embedder represents the embedding model.

[0111] Figure 7 A flowchart illustrating sample problem processing according to an embodiment of this application is shown schematically. Figure 8 A detailed flowchart illustrating the execution of a retrieval strategy according to an embodiment of this application is shown.

[0112] The following will combine Figure 5 , Figure 7 and Figure 8The above operation S220 will be described. In some embodiments, operation S220 specifically includes:

[0113] First, based on the problem handling prompt template indicated by the problem handling configuration parameters, the problem handling prompts are obtained.

[0114] According to an embodiment of this application, the problem handling configuration parameters indicate the execution status of the problem handling prompt word templates, that is, whether the problem handling rules corresponding to each problem handling prompt word template are executed when processing sample problem 701.

[0115] For example, the issue handling configuration parameters can be set by testers according to test requirements, and the issue handling prompt word template indicated in the issue handling configuration parameters can be used as the issue handling prompt word; the issue handling prompt word template can also be encapsulated as a JSON format configuration file, and when applied, the corresponding configuration file can be directly called in response to the issue handling prompt word template indicated by the issue handling configuration parameters.

[0116] Then, based on the problem processing rules indicated by the problem processing prompts, the preset large model is guided to process sample problem 701, resulting in multiple sub-sample problems 702.

[0117] According to an embodiment of this application, the problem handling rules may include at least one of problem rewriting, problem association, language type conversion, and no rules. The language type conversion may be Chinese to English conversion. If there are no problem handling rules, then the sample problem 701 will not be processed.

[0118] Here, the input for this step is a string-type question, and the output is a list-type question.

[0119] Furthermore, in order to remove redundant information and further improve the efficiency of problem processing, before processing sample problem 701, the following steps are also included: filtering keywords for sample problem 701, filtering words with high frequency but meaningless to the query. The keywords can be obtained by word segmentation and statistics of the pre-trained BERT model (a bidirectional encoder representation model based on Transformer). The input and output of this step are both string-type questions.

[0120] For example, problem handling can be represented as:

[0121] Q new i =[LLM(Prompt t (Query i) )], i = 0, 1, 2…;

[0122] Q new i =[LLM(Promptt+1 (Q new i :list))];

[0123] …

[0124] Q new i =[LLM(Prompt t+n (Q new i :list))];

[0125] Q new =[Q new i ];

[0126] Where [] indicates conversion to list type, Prompt t This represents the prompt template for the t-th issue. t Problem handling prompt template set {Prompt t Prompt t+1 …Prompt t+n}, n represents the number of problem-handling prompt word templates; Query represents the original problem, LLM represents the preset large model 505, i represents the number of problems, which will increase after problem expansion processing; Q new This represents a new list of questions, where each element is also a list.

[0127] Based on this, the corresponding problem-handling prompts can be quickly invoked using problem-handling configuration parameters to guide the preset large model to handle problems. By rewriting and expanding the problems, the quality of sample problem 701 can be improved, aligning with human intent and enhancing the comprehensiveness and accuracy of the retrieval.

[0128] Then, based on the data representation indicated by multiple retrieval strategies, the multiple subsample questions 702 are converted into multiple sets of retrieval questions.

[0129] According to embodiments of this application, the data representation format matches the retrieval type and may include at least one of keyword set, sparse vector, dense vector, structured metadata and knowledge graph, corresponding to keyword retrieval, sparse vector retrieval, dense vector retrieval, hybrid retrieval, keyword retrieval, filter retrieval and knowledge graph retrieval respectively; multiple subsample questions 702 are converted into a data representation format indicating the retrieval strategy to serve as the retrieval question.

[0130] For example, if the retrieval strategy indicates a sparse retrieval type, then the corresponding data representation is a sparse vector. A multi-question vectorizer 801 is used to convert multiple subsample questions 702 into sparse question vectors, also known as question vectors. The input to the normalized embedding model is a list of questions, and the output is a list of vectors. The multi-question vectorizer 801 can be an embedding model; it can be represented as:

[0131] V(Query new =text_embedder(Query new );

[0132] Where text_embedder represents the text embedding model; Query new This represents subsample problem 701; V(Query) new ) represents the transformed problem vector.

[0133] Then, multiple retrieval strategies are used to search the preset knowledge base 503 to obtain target authentication information that matches multiple sets of retrieval questions.

[0134] According to embodiments of this application, the retrieval types of multiple retrieval strategies may include at least one of dense retrieval, sparse retrieval, hybrid retrieval, keyword retrieval, filter retrieval, and knowledge graph retrieval.

[0135] For example, the retrieval tools included in each retrieval strategy are used to perform vector matching in a preset knowledge base 503, calculate the similarity between the question vector and the document block vector, obtain certified documents that match multiple sets of retrieval questions, calculate a retrieval score for each certified document, and output the multiple certified documents 601 with the highest retrieval scores as target certification information; the input to this step is a list-type vector, and the output is a list-type document; taking a combined retrieval containing multiple retrieval types as an example, the retrieval strategy can be represented as:

[0136] Docs t =Retrieval t (V(Query new ),DS);

[0137] Docs t+1 =Retrieval t+1 (V(Query new ),DS);

[0138] …

[0139] Docs t+n =Retrieval t+n (V(Query new ),DS);

[0140] Among them, Docs t This is the list of result documents from the t-th searcher, Docs t = m represents the number of documents; Retrieval represents the corresponding search engine, Retrieval∈{Retrieval t Retrieval t+1 Retrieval t+n} represents the retrieval strategy, n represents the number of retrieval machines, and DS represents the knowledge base.

[0141] The formula for calculating the search score can be expressed as:

[0142] FinalScore(doc ti )=W t ×score ti +W t+1 ×score t+1i +W t+2 ×score t+2i +...+W t+n ×score t+ni ;

[0143] Among them, W t +W t+1 +...+W t+n =1, W t It is the weight corresponding to the t-th searcher; doc ti It's Docs t The i-th document; score ti It is a certification document (doc). ti The search score; FinalScore (doc) ti ) is the retrieval score of certified document i.

[0144] The final search results can be represented as:

[0145] D sorted ={d(1),d(2),...,d(n)};

[0146] D final ={d(1),d(2),...,d(m)},m≥n;

[0147] Where d(i) represents the result according to FinalScore(doc) ti ) represents the i-th certified document in descending order, satisfying FinalScore(doc t1 )≥FinalScore(doc t2)≥...≥FinalScore(doc tn );D final It is a list of the top m certified documents with the highest scores.

[0148] In the above operations, each process can be executed by calling the configuration file, which improves retrieval efficiency.

[0149] Furthermore, when the retrieval configuration parameter 501 indicates that the re-ranking model 802 is enabled, the method further includes: enabling the re-ranking model 802 to re-rank the multiple output authentication documents, wherein the re-ranking model 802 can be a bge-reranker model, etc., and the input and output of this step are both list-type documents. The re-ranking process can be represented as:

[0150] Score rerank (d(i))=Rerank(d(i),Query), i∈{1,2,3,…,m};

[0151] D rerank ={dr(1),dr(2),...,dr(m)};

[0152] Where d(i) represents the i-th certified document after retrieval, d r (i) represents the i-th document after reordering, and satisfies dr(1)≥dr(2)≥...≥dr(m); D rerank It is a list of m reordered certified documents, where Query represents the original question and Rerank represents the preset reordering model 802.

[0153] Figure 9 A detailed flowchart illustrating a model question-and-answer format according to an embodiment of this application is shown.

[0154] Combination Figure 5 , Figure 9 In some embodiments, operation S230 includes: filtering multiple preset candidate prompt word templates based on the evaluation steps corresponding to multiple target authentication information 504 to obtain target prompt word templates; mapping the sample question, historical session information 502 and multiple target authentication information 504 to target prompt word templates 901 respectively to obtain multiple question-and-answer prompt words; and using a preset large model to process the multiple question-and-answer prompt words to obtain multiple responses to the sample question.

[0155] According to embodiments of this application, the evaluation step indicates the evaluation stage of the plurality of target authentication information. Each of the multiple candidate suggestion word templates has a matching retrieval evaluation step; that is, the suggestion word template set contains different suggestion word templates corresponding to different retrieval evaluation steps.

[0156] For example, the retrieval strategy evaluation method of this embodiment may include multiple evaluation stages, such as a retrieval stage, a question response stage, and a retrieval strategy evaluation stage. Based on this, in this step, the evaluation stage corresponding to the target authentication information is the question response stage. The prompt word template corresponding to the question response stage can be directly called to guide the preset large model to respond to questions, thereby improving the efficiency of the large model's response.

[0157] For example, multiple candidate prompt word templates can be encapsulated in a JSON format configuration file. The candidate prompt word template that matches the retrieval evaluation stage is called as the target prompt word template. The sample question 701, target authentication information, and historical session information (context) are mapped to the target prompt word template to obtain the question-and-answer prompt words. The question-and-answer prompt words are used to guide the preset large model to process the sample question based on the target authentication information and historical session information, generate the answer and the source of the answer, and output the response as the response to the sample question.

[0158] Here, the input to this operation can be a string-type prompt, and the output can be a string-type answer and a list-type document source.

[0159] For example, the target prompt template could be: "Based on the following information, please answer the question concisely in one or two sentences. Note that the answer must use the same language as the document."

[0160] Use only the relevant details provided in the documentation.

[0161] Support your answer with brief, verbatim quotations from the most relevant documents.

[0162] Notice:

[0163] Some documents may be irrelevant to the question.

[0164] There is no need to cite the source in your answer.

[0165] Context: {}

[0166] document:{}

[0167] question:{}

[0168] answer:".

[0169] For example, this operation can be represented as:

[0170] Prompt=(history, D rerank (contexti),Enrich_llm(metai),Query), i∈{1,2,3,…,m};

[0171] Answer = LLM(Prompt);

[0172] Docs=D rerank ;

[0173] In the formula, LLM represents the preset large model 505, Answer represents the string type answer, Docs represents the source document of the answer, history represents the context, and Enrich_llm(metai) represents the polished result.

[0174] In the embodiments of this application, a prompt word template is invoked based on the current evaluation stage. The prompt word template matching the evaluation stage is used to generate question and answer prompt words, thereby improving the generation speed of question and answer prompt words. The question and answer prompt words are used to guide the large model to respond to the sample question 701 within the scope of authentication domain knowledge that matches the sample question 701, thereby improving the efficiency and accuracy of the large model's question and answer.

[0175] In some embodiments, based on multiple responses, the retrieval strategies corresponding to each of the multiple target authentication information are evaluated to obtain multiple evaluation results for the multiple retrieval strategies, including: filtering multiple preset candidate evaluation prompts based on the retrieval types indicated by the multiple retrieval strategies to obtain multiple target evaluation prompts; and guiding a preset large model to process multiple target authentication information, multiple responses, and a preset standard answer based on the evaluation rules indicated by each of the multiple target evaluation prompts to obtain multiple evaluation results for the retrieval strategies.

[0176] According to embodiments of this application, each of the multiple candidate evaluation prompts has a matching search type.

[0177] Based on this, different evaluation prompts can be selected according to the search type to evaluate the search strategy and improve the accuracy of the evaluation.

[0178] For example, in the case of a mixed search, candidate evaluation prompts may include "Please evaluate whether the response integrates information from multiple sources, balancing relevance and diversity. Focus on whether the information coverage is comprehensive and the structure is clear." In the case of a sparse search, candidate evaluation prompts may include "Please evaluate whether the response covers the keywords in the sample question, whether it contains key terms, and whether there are any missing information." In the case of a dense search, candidate evaluation prompts may include "Please evaluate whether the response is semantically highly relevant to the sample question, whether it deeply understands the intent of the sample question, and whether it has made appropriate semantic expansions."

[0179] In some embodiments, based on multiple responses, the retrieval strategies corresponding to each of the multiple target authentication information are evaluated to obtain multiple evaluation results for the multiple retrieval strategies, including:

[0180] First, based on the degree of matching between multiple target authentication information and preset standard answers, multiple process evaluation scores are obtained.

[0181] According to embodiments of this application, the process evaluation score indicates the retrieval effectiveness of the corresponding retrieval strategy.

[0182] The target authentication information includes multiple text segments. Further, for each retrieval strategy, firstly, using pre-set relevance assessment prompts, a pre-set large model is guided to perform relevance analysis on the multiple text segments and the pre-set standard answer to obtain relevant and irrelevant text segments; then, based on the ratio of the number of relevant text segments to the total number of multiple text segments, the recall precision is obtained; then, based on the ratio of the number of correct statements contained in the relevant text segments to the total number of statements in the pre-set standard answer, the recall rate is obtained; finally, the recall precision and recall rate are combined to obtain the process evaluation score.

[0183] Here, "declaration" refers to the triplet information obtained by extracting entity relations from the input text through a preset large model. "Correct declaration" refers to the declaration in the preset standard answer contained in the relevant text segment, and "missing declaration" refers to the declaration in the preset standard answer that does not appear in the relevant text segment. "Number of relevant text segments" represents the number of text segments recalled that are related to the preset standard answer.

[0184] For example, the process evaluation score F1 can be expressed as:

[0185] F1 = 2 * (Recall * Precision) / (Recall + Precision);

[0186] Recall = Number of correct statements in the relevant text segment / Total number of statements in the default standard answer;

[0187] Recall precision = Number of relevant text segments / Total number of multiple text segments;

[0188] Based on this, recall precision characterizes the retrieval accuracy of the retrieval strategy, while recall rate characterizes the retrieval comprehensiveness of the retrieval strategy. By integrating recall precision and recall rate, the execution process of the retrieval strategy can be comprehensively evaluated to ensure the accuracy of the evaluation.

[0189] Then, based on the degree of matching between multiple target authentication information and the preset standard answer, as well as the degree of matching between multiple responses and the preset standard answer, multiple result evaluation scores are obtained respectively.

[0190] According to embodiments of this application, the result evaluation score indicates the degree to which the corresponding target authentication information was used when generating the response.

[0191] Furthermore, for each retrieval strategy, firstly, fidelity is obtained based on the ratio of the sum of the number of correct claims in relevant text segments to the sum of the number of incorrect claims in multiple text segments and the sum of the number of correct and incorrect claims; then, context utilization is obtained based on the ratio of the number of correct claims in relevant text segments to the sum of the number of correct and missing claims in relevant text segments; finally, fidelity and context utilization are used as the result evaluation score.

[0192] Here, an incorrect statement refers to a statement in the relevant text segment that is neither a correct statement nor a missing statement; that is, a triplet that is not present in the default standard answer but is recalled.

[0193] For example, loyalty can be expressed as:

[0194] Fidelity = (Number of correct statements in the relevant text segment + Number of incorrect statements across multiple text segments) / (Number of correct statements + Number of incorrect statements);

[0195] Context utilization can be expressed as:

[0196] Context utilization = number of correct declarations in the relevant text segment / (number of correct declarations in the relevant text segment + number of missing declarations in the relevant text segment).

[0197] Then, the process evaluation score and the result evaluation score are combined to obtain the evaluation result for the retrieval strategy.

[0198] For example, the process evaluation score and the result evaluation score are weighted and summed to obtain the evaluation score for the retrieval strategy.

[0199] Based on this, the evaluation strategy is comprehensively evaluated from both process and outcome dimensions.

[0200] Furthermore, the most effective retrieval strategy can be selected based on the above evaluation results to reduce the scope of optimization and provide positive feedback for the next retrieval strategy formulation. Alternatively, the selection scope covered by multiple candidate evaluation rule templates can be narrowed based on the above evaluation results or prior knowledge. For example, the candidate evaluation rule template corresponding to the retrieval strategy with the lowest evaluation score can be eliminated to narrow the selection scope. During the selection, the candidate evaluation rule template or retrieval configuration parameter 501 corresponding to the retrieval strategy with the highest historical evaluation score is preferred.

[0201] Furthermore, in the above operations, the embodiments of this application specify the input and output formats of data in each operation. Based on the above specifications, the data between various operations can be quickly connected, and it is also convenient to quickly integrate other operations or modules; different operations are used to implement different operating logics, thereby achieving decoupling of the operating logic between different operations.

[0202] Based on the above-described retrieval strategy evaluation method, this application also provides a retrieval strategy evaluation device. The following will be combined with... Figure 10 The device is described in detail.

[0203] Figure 10 A schematic block diagram of a retrieval strategy evaluation apparatus according to an embodiment of this application is shown.

[0204] like Figure 10 As shown, the retrieval and evaluation device 1000 of this embodiment includes an acquisition module 1010, a retrieval module 1020, a question-and-answer module 1030, an evaluation module 1040, and a filtering module 1050.

[0205] The acquisition module 1010, in response to receiving a search strategy update instruction, combines multiple search rules that match the search configuration parameters to obtain multiple search strategies. The search configuration parameters indicate the search type of the multiple search strategies and the search conditions that match the search type. In one embodiment, the acquisition module 1010 can be used to perform the operation S210 described above, which will not be repeated here.

[0206] The retrieval module 1020 is used to search a preset knowledge base based on the sample question using multiple retrieval strategies to obtain multiple target authentication information, which indicates the authentication domain knowledge that matches the sample question. In one embodiment, the retrieval module 1020 can be used to perform the operation S220 described above, which will not be repeated here.

[0207] The question-answering module 1030 is used to process multiple target authentication information based on sample questions using a preset large model, and output multiple responses to the sample questions. In one embodiment, the question-answering module 1030 can be used to perform the operation S230 described above, which will not be repeated here.

[0208] The evaluation module 1040 is used to evaluate the retrieval strategies corresponding to each of the multiple target authentication information based on multiple responses, and obtain multiple evaluation results for the multiple retrieval strategies. In one embodiment, the evaluation module 1040 can be used to perform the operation S240 described above, which will not be repeated here.

[0209] The filtering module 1050 is used to filter multiple retrieval strategies based on multiple evaluation results to obtain a target retrieval strategy. The target retrieval strategy is then used as the updated retrieval strategy to respond to the received question. The target retrieval strategy is then used to search in a preset knowledge base to obtain authentication domain knowledge that matches the question.

[0210] According to an embodiment of this application, the acquisition module 1010 includes a filtering submodule, a generation submodule, a mapping submodule, and a combination submodule. The filtering submodule is used to filter multiple preset candidate search rule templates based on the search type indicated by the search configuration parameters to obtain a target search rule template. The target search rule template indicates the hierarchical data structure corresponding to multiple search rules. The generation submodule is used to generate multiple candidate values ​​for the search configuration parameters based on the numerical type corresponding to the search configuration parameters and at least one of the parameter value ranges matching the numerical type. The numerical type includes at least one of integer, floating-point number, boolean value, string, range parameter, and no range parameter. The mapping submodule is used to combine multiple candidate values ​​to obtain multiple candidate value sets, and to map the multiple candidate value sets 307 to the search rule templates respectively to obtain multiple search rules. The combination submodule is used to combine multiple search rules to obtain multiple search strategies.

[0211] According to an embodiment of this application, the generation submodule is specifically used to filter from a plurality of preset candidate value generation rules based on the value type to obtain a target value generation rule, wherein each of the plurality of candidate value generation rules has a matching value type; and to generate a plurality of candidate values ​​for the retrieval configuration parameters within the parameter value range using the target value generation rule, wherein the target value generation rule is a full traversal rule, a switch-type value rule, an integer step size traversal rule, a floating point step size traversal rule, a single selection rule, a multiple selection rule, or a no-selection rule.

[0212] According to an embodiment of this application, the retrieval module 1020 is specifically used to obtain problem processing prompt words based on the problem processing prompt word template indicated by the problem processing configuration parameters; guide a preset large model to process sample problems based on the problem processing rules indicated by the problem processing prompt words to obtain multiple sub-sample problems, the problem processing rules including at least one of problem rewriting, problem association, and language type conversion; convert the multiple sub-sample problems into multiple sets of retrieval problems based on the data representation forms indicated by multiple retrieval strategies, the data representation forms including at least one of keyword sets, sparse vectors, dense vectors, structured metadata, and knowledge graphs; and use multiple retrieval strategies to search in a preset knowledge base to obtain target authentication information matching the multiple sets of retrieval problems, the retrieval types of the multiple retrieval strategies including at least one of dense retrieval, sparse retrieval, hybrid retrieval, keyword retrieval, filter retrieval, and knowledge graph retrieval.

[0213] According to an embodiment of this application, the question-answering module 1030 is specifically used to filter multiple preset candidate prompt word templates based on the retrieval evaluation steps corresponding to multiple target authentication information to obtain target prompt word templates. Each of the multiple candidate prompt word templates has a matching evaluation step, and the evaluation step indicates the evaluation stage of the multiple target authentication information. The sample question, historical conversation information, and multiple target authentication information are mapped to the target prompt word templates respectively to obtain multiple question-answering prompt words. The multiple question-answering prompt words are processed using a preset large model to obtain multiple responses to the sample question.

[0214] According to an embodiment of this application, the evaluation module 1040 is specifically used to filter multiple preset candidate evaluation prompts based on the search types indicated by multiple search strategies, thereby obtaining multiple target evaluation prompts, each of which has a matching search type; based on the evaluation rules indicated by each of the multiple target evaluation prompts, the module guides a preset large model to process multiple target authentication information, multiple responses, and preset standard answers, thereby obtaining multiple evaluation results for the search strategies.

[0215] According to an embodiment of this application, the evaluation module 1040 is specifically used to obtain multiple process evaluation scores based on the degree of matching between multiple target authentication information and preset standard answers, wherein the process evaluation scores indicate the retrieval effectiveness of the corresponding retrieval strategy; to obtain multiple result evaluation scores based on the degree of matching between multiple target authentication information and preset standard answers and the degree of matching between multiple responses and preset standard answers, wherein the result evaluation scores indicate the degree of use of the corresponding target authentication information when generating responses; and to merge the multiple process evaluation scores and the multiple answer evaluation scores to obtain multiple evaluation results for multiple retrieval strategies.

[0216] According to embodiments of this application, any multiple modules among the acquisition module 1010, retrieval module 1020, question-answering module 1030, and evaluation module 1040 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the acquisition module 1010, retrieval module 1020, question-answering module 1030, and evaluation module 1040 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the acquisition module 1010, retrieval module 1020, question-answering module 1030, and evaluation module 1040 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0217] Figure 11 A block diagram schematically illustrates an electronic device suitable for implementing a retrieval strategy evaluation method according to an embodiment of this application.

[0218] like Figure 11 As shown, an electronic device 1100 according to an embodiment of this application includes a processor 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage portion 1108 into a random access memory (RAM) 1103. The processor 1101 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1101 may also include onboard memory for caching purposes. The processor 1101 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0219] RAM 1103 stores various programs and data required for the operation of electronic device 1100. Processor 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Processor 1101 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 1102 and / or RAM 1103. It should be noted that the programs may also be stored in one or more memories other than ROM 1102 and RAM 1103. Processor 1101 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0220] According to embodiments of this application, the electronic device 1100 may further include an input / output (I / O) interface 1105, which is also connected to a bus 1104. The electronic device 1100 may also include one or more of the following components connected to the input / output (I / O) interface 1105: an input section 1111 including a keyboard, mouse, etc.; an output section 1107 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN card, modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the input / output (I / O) interface 1105 as needed. A removable medium 1111, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1110 as needed so that computer programs read from it can be installed into the storage section 1108 as needed.

[0221] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0222] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM1102 and / or RAM1103 and / or one or more memories other than ROM1102 and RAM1103 described above.

[0223] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the retrieval strategy evaluation method provided in the embodiments of this application.

[0224] When the computer program is executed by the processor 1101, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0225] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1109, and / or installed from the removable medium 1111. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0226] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1109, and / or installed from the removable medium 1111. When the computer program is executed by the processor 1101, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0227] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0228] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0229] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A method for evaluating retrieval strategies, characterized in that, The method includes: In response to receiving a search strategy update instruction, multiple search rules that match the search configuration parameters are combined to obtain multiple search strategies. The search configuration parameters indicate the search type of the multiple search strategies and the search conditions that match the search type. Based on the sample question, the multiple retrieval strategies are used to search in a preset knowledge base to obtain multiple target authentication information, which indicates authentication domain knowledge that matches the sample question. The multiple target authentication information are processed using a pre-defined large model based on the sample question, and multiple responses are output for the sample question. Based on the multiple responses, the retrieval strategies corresponding to each of the multiple target authentication information are evaluated to obtain multiple evaluation results for the multiple retrieval strategies; Based on the multiple evaluation results, a target retrieval strategy is obtained by filtering among the multiple retrieval strategies. The target retrieval strategy is then used as the updated retrieval strategy to respond to a received question by searching the preset knowledge base to obtain authentication domain knowledge that matches the question.

2. The method according to claim 1, characterized in that, The combination of multiple search rules matching the search configuration parameters to obtain multiple search strategies includes: Based on the search type indicated by the search configuration parameters, multiple preset candidate search rule templates are filtered to obtain a target search rule template, which indicates the hierarchical data structure corresponding to the multiple search rules; Based on at least one of the numerical type corresponding to the search configuration parameter and the parameter value range matching the numerical type, multiple candidate values ​​of the search configuration parameter are generated. The numerical type includes at least one of integer, floating-point number, boolean value, string, range parameter and no range parameter. The multiple candidate values ​​are combined to obtain multiple candidate value sets; The multiple candidate value sets are mapped to the retrieval rule templates respectively to obtain multiple retrieval rules; By combining the multiple search rules, multiple search strategies can be obtained.

3. The method according to claim 2, characterized in that, The step of generating multiple candidate values ​​for the search configuration parameters based on at least one of the parameter type and parameter value range corresponding to the search configuration parameters includes: Based on the numerical type, a target numerical generation rule is obtained by filtering from a plurality of preset candidate numerical generation rules, wherein each of the plurality of candidate numerical generation rules has a matching numerical type. The target value generation rule is used to generate multiple candidate values ​​for the retrieval configuration parameter within the parameter value range. The target value generation rule can be a full traversal rule, a switch-type value rule, an integer step size traversal rule, a floating-point step size traversal rule, a single selection rule, a multiple selection rule, or a no-selection rule.

4. The method according to claim 1, characterized in that, The process of retrieving multiple target authentication information from a preset knowledge base based on a sample question using the multiple retrieval strategies includes: Based on the problem handling prompt template indicated by the problem handling configuration parameters, the problem handling prompt is obtained; Based on the problem processing prompts indicated by the problem processing rules, the preset large model is guided to process the sample problem to obtain multiple sub-sample problems. The problem processing rules include at least one of problem rewriting, problem association, and language type conversion. Based on the data representation forms indicated by the multiple retrieval strategies, the multiple subsample questions are converted into multiple sets of retrieval questions, and the data representation forms include at least one of keyword sets, sparse vectors, dense vectors, structured metadata, and knowledge graphs; The multiple retrieval strategies are used to search the preset knowledge base to obtain target authentication information that matches the multiple sets of retrieval questions. The retrieval types of the multiple retrieval strategies include at least one of dense retrieval, sparse retrieval, hybrid retrieval, keyword retrieval, filter retrieval, and knowledge graph retrieval.

5. The method according to claim 1, characterized in that, The process of using a pre-defined large model to process the multiple target authentication information based on the sample question, and outputting multiple responses to the sample question, includes: Based on the evaluation steps corresponding to the multiple target authentication information, multiple preset candidate prompt word templates are filtered to obtain target prompt word templates. Each of the multiple candidate prompt word templates has a matching evaluation step, and the evaluation step indicates the evaluation stage of the multiple target authentication information. The sample question, historical session information, and multiple target authentication information are respectively mapped to the target prompt word template to obtain multiple question-and-answer prompt words; The preset large model is used to process the multiple question-and-answer prompts to obtain multiple responses to the sample question.

6. The method according to claim 1, characterized in that, The step of evaluating the retrieval strategies corresponding to the multiple target authentication information based on the multiple responses, and obtaining multiple evaluation results for the multiple retrieval strategies, includes: Based on the search types indicated by the multiple search strategies, multiple preset candidate evaluation prompts are filtered to obtain multiple target evaluation prompts, each of which has a matching search type. Based on the evaluation rules indicated by the multiple target evaluation prompts, the preset large model is guided to process the multiple target authentication information, the multiple responses, and the preset standard answer to obtain multiple evaluation results for the retrieval strategy.

7. The method according to claim 1, characterized in that, The step of evaluating the retrieval strategies corresponding to the multiple target authentication information based on the multiple responses, and obtaining multiple evaluation results for the multiple retrieval strategies, includes: Based on the degree of matching between the multiple target authentication information and the preset standard answer, multiple process evaluation scores are obtained, and the process evaluation scores indicate the retrieval effectiveness of the corresponding retrieval strategy; Based on the degree of matching between the multiple target authentication information and the preset standard answer, and the degree of matching between the multiple responses and the preset standard answer, multiple result evaluation scores are obtained respectively. The result evaluation scores indicate the degree of use of the corresponding target authentication information when generating the response. The evaluation scores of the multiple processes and the evaluation scores of the multiple answers are combined to obtain multiple evaluation results for the multiple retrieval strategies.

8. The method according to any one of claims 1 to 7, characterized in that, The preset knowledge base includes multiple candidate authentication information, and the preset knowledge base is obtained using the following method: Based on the document processing prompt template indicated by the document processing configuration parameters, document processing prompts are obtained. Based on the document processing rules indicated by the document processing prompt words, the preset large model is guided to process multiple certified documents respectively, obtain multiple metadata of the multiple certified documents, and associate the multiple metadata with the corresponding certified documents respectively. The document processing rules include at least one of extracting document keywords and summarizing document content, and the metadata includes at least one of summary and keywords. The text data in the multiple certified documents is divided into multiple text blocks. The multiple text blocks are vectorized using a pre-trained embedding model. The resulting document vectors are used as candidate certification information to form a preset knowledge base.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

Images