Search engine performance evaluation method and system based on large language model agent
By combining large-scale language model agents with the Cranfield evaluation system, and utilizing the BM25F algorithm and multi-agent debate framework, the accuracy and applicability issues of existing search engine performance evaluations are solved, achieving efficient and accurate search engine performance evaluation.
Patent Information
- Application Number
- CN202510860401.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-25
- Publication Date
- 2025-11-07
AI Technical Summary
Existing search engine performance evaluation methods are difficult to accurately simulate human evaluation, have high costs and limitations, and are difficult to adapt to diverse user needs and search scenarios.
By combining large-scale language model agents with the Cranfield evaluation system, and through prompting engineering and a multi-agent debate framework, we simulate the relevance labeling of human search results, use the BM25F algorithm to calculate word frequency similarity, and guide agents to play different user roles in debate.
It achieves more accurate and reliable search engine performance evaluation, reduces the bias and instability of large language models, and improves the consistency and applicability of the evaluation.
Smart Images
Figure CN120909896A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence, in particular to a search engine performance evaluation method and system based on a large language model agent. BACKGROUND
[0002] Under the background of information explosion, search engines have become an indispensable Internet infrastructure, deeply affecting information acquisition, business activities, and even daily life. Currently, search engine technology is becoming more mature and showing a trend of diversification. From the initial keyword matching to semantic search, knowledge graph, artificial intelligence and machine learning application, it more accurately understands user intent and provides more intelligent search results.
[0003] With the continuous expansion of the influence of search engines, many challenges and controversies are also faced. Search engines still have many retrieval results that are not related to the user's query intent. This not only fails to meet the user's needs, but may even contain misleading or incorrect information.
[0004] In addition, search engines, in addition to being an information tool for ordinary users, are also important digital marketing tools and business platforms. The business model of search engines mainly relies on advertising revenue, through displaying advertisements related or unrelated to search keywords, manipulating search results, bringing huge benefits to search engine companies. These factors outside the technology have greatly affected the user experience of search engines, and many users have questioned and criticized the various problems of search engines.
[0005] Currently, with the explosive growth of Internet information, search engine products are constantly updated, and user query needs are constantly changing. In this highly dynamic environment, automated search engine performance evaluation becomes increasingly important. The task of search engine performance evaluation is to measure and evaluate the effectiveness of search engines in meeting user information needs.
[0006] Since search engines involve indexing, query processing, search, ranking, feedback, and iteration, they involve many algorithms and technologies. Moreover, search engine products in the industry use many experiences summarized from engineering practice and user data. In addition, under the existing business model, the algorithms and technologies behind search engines are highly confidential. Therefore, researchers often cannot check the internal code and logic of search engines. In this case, search engine performance evaluation can be said to be a black box test of search engines.
[0007] Search engine performance evaluation research can be traced back to early research in the field of information retrieval, especially the Cranfield evaluation system in the United Kingdom in the 1960s. The core of the Cranfield evaluation system is to conduct relevance evaluation manually, and its flowchart is as followsFigure 1 Manual evaluation is considered as the "gold standard" or ground truth for building information retrieval system evaluation, as humans can understand the nuances of information needs, the context of document content, and potential relevance.
[0008] However, manual relevance evaluation requires a large amount of human and time cost, which is obviously no longer applicable in today's rapidly changing Internet environment, which is the limitation of the Cranfield evaluation system. Despite this, it is still the basic method for the academic and industrial communities to evaluate the performance of information retrieval systems. It has undergone extensive theoretical and practical verification and provides a standardized experimental framework.
[0009] To reduce the evaluation cost, researchers have proposed many automated search engine performance evaluation methods. However, the existing methods still have many limitations. These methods are difficult to fully simulate the complexity of human evaluation, and are prone to inaccurate evaluation due to simplified models and data bias. In addition, many methods rely on specific data or scenarios, making it difficult to effectively generalize to different search engines, user groups, and diverse search needs. These limitations collectively constrain the application of current automatic evaluation methods in practice.
[0010] Large language models are a key technology that has made breakthrough progress in the field of artificial intelligence, especially in natural language processing, in recent years. Large language model agents are considered to be the foundation of artificial intelligence applications. They are entities driven by large language models that can plan and take actions, or perform goals through multiple iterations to solve specific tasks. Because of the high consistency between the understanding and language ability of large language models and humans, large language model technology is very suitable for combining with the Cranfield evaluation system. The present invention uses large language model agents to simulate humans reading title, abstract and URL information to achieve more accurate and universal retrieval result relevance labeling than previous related work. SUMMARY
[0011] The main purpose of the present invention is to propose a search engine performance evaluation method and system based on large language model agents, which combines large language model technology with the classic Cranfield evaluation system, and uses the natural language understanding ability and knowledge background of large language model agents to more accurately simulate humans to perform relevance labeling of retrieval results.
[0012] To achieve the above invention purpose, the technical scheme of the present invention includes the following contents.
[0013] A search engine performance evaluation method based on large language model agents, the method comprising:
[0014] Step S1: performing a search of a query set in a search engine to be evaluated to obtain a search result set; wherein the query set contains at least one query, and the search result set contains a search result of the query;
[0015] Step S2: defining N roles of agents, wherein the agents are constructed based on a large language model, and the large language model has injected artificial relevance annotation experience in a prompt template;
[0016] Step S3: calculating a similarity between the query and the search result;
[0017] Step S4: injecting a role description of the agent, the query, the search result, the similarity, and a historical annotation result of the search result by the agent and a reason for the historical annotation result into the prompt template to obtain a current annotation result of the search result by the agent and a reason for the current annotation result;
[0018] Step S5: after updating the historical annotation result and the reason for the historical annotation result based on the current annotation result and the reason for the current annotation result, re-executing Step S4, and obtaining a final annotation result of the search result by the agent after the re-execution reaches a set value;
[0019] Step S6: synthesizing the final annotation results of the search result by the agents to obtain a performance evaluation result of the search engine to be evaluated.
[0020] Further, the roles of the agents include the public, the experts, and the analysts.
[0021] Further, the search result includes the title, the abstract, and the URL.
[0022] Further, the similarity between the query and the search result is calculated, including:
[0023] calculating a similarity S1 between the title and the query based on the BM25F algorithm;
[0024] calculating a similarity S2 between the abstract and the query based on the BM25F algorithm;
[0025] weighting and summing the similarity S1 and the similarity S2 to obtain the similarity between the query and the search result.
[0026] Further, when the similarity S1 and the similarity S2 are weighted and summed, the weight corresponding to the similarity S1 is greater than the weight corresponding to the similarity S2.
[0027] Further, synthesizing the final annotation results of the search result by the agents to obtain the performance evaluation result of the search engine to be evaluated includes:
[0028] The final relevance annotation result of the search result is obtained by integrating the final annotation results of the search result of each agent through voting or averaging.
[0029] The performance evaluation result of the search engine to be evaluated is obtained by performing statistics and analysis based on the final relevance annotation results of all search results.
[0030] A search engine performance evaluation system based on large language model agents, the system comprising:
[0031] A data acquisition module for performing search of a query set in a search engine to be evaluated to obtain a search result set; wherein the query set contains at least one query, and the search result set contains the search result of the query;
[0032] A role definition module for defining the roles of N agents, the agents being constructed based on a large language model, and the prompt module of the large language model having injected artificial relevance annotation experience;
[0033] A similarity calculation module for calculating the similarity between the query and the search result;
[0034] A search result annotation module for injecting the role description of the agent, the query, the search result, the similarity, and the historical annotation result of the search result of the agent and the reason for the historical annotation result into the prompt template to obtain the current annotation result of the search result of the agent and the reason for the current annotation result; after updating the historical annotation result and the reason for the historical annotation result based on the current annotation result and the reason for the current annotation result, re-executing the search result annotation, and obtaining the final annotation result of the search result of the agent after the re-execution times reach a set value.
[0035] An evaluation result generation module for integrating the final annotation results of the search result of each agent to obtain the performance evaluation result of the search engine to be evaluated.
[0036] An electronic device comprising a processor and a memory storing computer program instructions; the processor executes the computer program instructions to implement the search engine performance evaluation method based on large language model agents according to any one of the above.
[0037] A computer readable storage medium, characterized in that the computer readable storage medium stores computer program instructions, and the computer program instructions are executed by a processor to implement the search engine performance evaluation method based on large language model agents according to any one of the above.
[0038] A computer program product, characterized in that when the computer program product runs on a computer device, the computer device is caused to execute the search engine performance evaluation method based on the large language model intelligent agent of any one of the above.
[0039] Compared with the prior art, the present application has at least the following beneficial effects.
[0040] Although the use of large language model technology can simulate human relevance labeling of retrieval results to a large extent, it also leads to some defects of large language model being brought into the evaluation. For example, large language model may produce hallucinations or have certain preferences during the evaluation, resulting in deviation of the evaluation. In this case, the evaluation result may only have reference value for a small part of users. In addition, since the large language model is a probability-based generative model, repeated relevance labeling of the same retrieval result using a large language model may be correct sometimes and incorrect sometimes, especially for some retrieval results that are difficult to judge directly. This instability will also lead to instability of the evaluation result. Therefore, the present application improves the consistency with human evaluation on the basis of fusing large language model technology and Cranfield evaluation system through prompt engineering and multi-agent debate framework and other related technologies.
[0041] Prompt engineering can guide the large language model to understand the user's intention, stimulate its internal knowledge, and finally generate more accurate, reliable, and expected high-quality output. The present application first uses prompt engineering technology to guide the intelligent agent to perform relevance labeling of retrieval results from a more scientific perspective and reduce hallucinations. Researchers have pointed out that the user's web search intention can be divided into three categories: navigation, information and transaction search. For different types of search, the criteria for determining whether the search result is relevant to the query are different, and correctly determining the type of search intention helps to more accurately determine whether the search result is relevant to the query. The present application injects these artificial relevance labeling experiences into the prompt, thereby guiding the intelligent agent to perform relevance labeling of retrieval results according to more scientific criteria.
[0042] Meanwhile, the present application calculates the term frequency similarity between the retrieval result text and the query through the BM25F algorithm, and injects this information into the agent as reference information for relevance labeling. The present application considers that the keyword frequency in the retrieval result text can also reflect the relevance of the retrieval result and the query to a certain extent. If a retrieval result does not contain query keywords at all, then this retrieval result is likely to be irrelevant to the query. Moreover, the algorithm and formula of BM25F are fixed and have no randomness, and can stably calculate the keyword frequency in the retrieval result text, thus helping to reduce the deviation and instability brought by the large language model, and preventing the misjudgment of the agent. The schematic diagram for prompting the large language model agent to guide the relevance labeling of the retrieval result is shown in Figure 2 .
[0043] In addition, in order to generate evaluation results more in line with artificial evaluation standards, the present application further introduces a multi-agent debate framework to simulate the artificial evaluation process, and uses multiple agents to play the roles of different groups of users to take turns speaking, promotes the agent to think more deeply by colliding different opinions in the debate, and makes the relevance labeling task benefit from a more comprehensive perspective. The user roles can include different types such as the public, experts and analysts, which have different emphases for the relevance labeling task. In addition, each role is assigned a name to facilitate better communication among agents in the debate. By letting the agent play different roles, not only can different opinions be complemented to deal with the complexity of the relevance labeling task from more dimensions, but also the understanding of the agent for the task context can be promoted, and more in-depth independent thinking can be guided. BRIEF DESCRIPTION OF DRAWINGS
[0044] Figure 1 is a flowchart of the Cranfield evaluation system.
[0045] Figure 2 is a schematic diagram for the agent to label the relevance of the retrieval result.
[0046] Figure 3 is a schematic diagram for the multi-agent to label the relevance of the retrieval result. DETAILED DESCRIPTION
[0047] In order to enable personnel in the technical field to better understand the technical solutions in the embodiments of the present application, and to make the purpose, features and advantages of the present application more apparent and easy to understand, the present application is further described in detail below in conjunction with the drawings and embodiments.
[0048] The search engine performance evaluation method of the present application is based on the Cranfield evaluation of multi-agent debate. Since the Cranfield evaluation system is suitable for various search scenarios, it can comprehensively reflect the performance of the search engine in meeting various needs. It can not only evaluate the performance of a single search engine, but also compare the performance between multiple search engines. Therefore, the present application first selects the Cranfield evaluation system as the basis of the evaluation method. The core of the Cranfield method is to mark the relevance of the search results by artificial means. In the process of artificial marking, the final judgment can be made by reading the title, abstract and URL text information of the search results, which involves the natural language understanding ability of human beings and the background knowledge related to the query demand. Since the natural language understanding ability of large language models has high consistency with human beings, and through pre-training on a large amount of human data, it has a rich knowledge background, even more comprehensive than general human beings. Therefore, the present application combines large language model technology and Cranfield evaluation system to realize more accurate and universal relevance marking of search results.
[0049] Specifically, the search engine performance evaluation method based on large language model agent of the present application, as shown in Figure 2 , includes the following steps.
[0050] Step 001: Define a query set. A user defines a query set Queries according to needs in advance, which contains at least one query word.
[0051] Step 002: Get search results. Perform search in all search engines to be evaluated to obtain the search result set Results corresponding to the query set Queries.
[0052] Step 003: Large language model agent initialization. Define N kinds of agent roles, such as public, expert or analyst, etc.
[0053] Step 004: Multi-agent debate parameter initialization. Define the number of multi-agent debate rounds M. Define the variable history to record the debate history of the agent.
[0054] Step 005: Large language model prompt template initialization. Pre-inject artificial relevance marking experience into the template, and at the same time constrain the JSON format of the agent response in the template, and reserve slots for variables.
[0055] Step 006: Calculate BM25F similarity. For one query query in Queries, a certain search engine's search result corresponding to query is result. result contains information such as title, abstract, and URL. Use the BM25F algorithm to calculate the similarity between the title and abstract fields and query. Among them, compared with the abstract field, a higher weight should be set for the title field.
[0056] Step 007: Relevance annotation. For each search result result, use an agent Agent n of a certain role to perform relevance annotation. Inject the role description, query, result, BM25F similarity, and history into the prompt template, and call a large language model to annotate the relevance of the search result. Agent n answers the annotation result label n and its reason reason n according to the pre-defined JSON format in the prompt template. Among them, label n is a number representing the relevance level of the search result, and reason n is a string. At the same time, add Agent n 's label n and reason n to history.
[0057] Step 008: Multi-agent debate. N different agents perform relevance annotation on result according to step 007 in turn, and cycle for M rounds. Each time the annotation result and reason need to be updated in history.
[0058] Step 009: Comprehensive annotation result. Take the annotation results of N different agents in the last round, and combine the annotation results of multiple agents into one result through voting or averaging, etc. as the final relevance annotation result of result. Figure 3 A schematic diagram for performing relevance annotation on search results through multi-agent debate.
[0059] Step 010: Statistics and analysis. After performing relevance annotation on all search results in Results, use the annotation results to calculate various performance evaluation indicators of the search engine on Queries, such as average accuracy, and perform analysis and comparison.
[0060] Finally, it should be pointed out that the above-mentioned embodiments are only used to illustrate the technical solutions of the present application but not to limit the present application. Although the present application has been described in detail by using the embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced, without departing from the spirit and scope of the present application, and all these modifications and equivalents should be included in the scope of the claims of the present application.
Claims
1. A method for evaluating the performance of a search engine based on a large language model agent, characterized in that, The method comprises: Step S1: performing a search of a query set in a search engine to be evaluated to obtain a search result set; wherein the query set contains at least one query, and the search result set contains a search result of the query; Step S2: defining N roles of agents, wherein the agents are constructed based on a large language model, and the large language model has injected artificial relevance annotation experience into a prompt template thereof; Step S3: calculating a similarity between the query and the search result; Step S4: injecting a role description of the agent, the query, the search result, the similarity, and a historical annotation result of the search result by the agent and a reason for the historical annotation result into the prompt template to obtain a current annotation result of the search result by the agent and a reason for the current annotation result; Step S5: after updating the historical annotation result and the reason for the historical annotation result based on the current annotation result and the reason for the current annotation result, re-executing Step S4, and obtaining a final annotation result of the search result by the agent after a re-execution number reaches a set value; Step S6: synthesizing the final annotation results of the search result by the agents to obtain a performance evaluation result of the search engine to be evaluated.
2. The method of claim 1, wherein, The roles of the agents include a public, an expert, and an analyst.
3. The method of claim 1, wherein, The search result includes a title, an abstract, and a URL.
4. The method of claim 3, wherein, The similarity between the query and the search result is calculated, comprising: calculating a similarity S1 between the title and the query based on a BM25F algorithm; calculating a similarity S2 between the abstract and the query based on the BM25F algorithm; weighting and summing the similarity S1 and the similarity S2 to obtain the similarity between the query and the search result.
5. The method of claim 4, wherein, When the similarity S1 and the similarity S2 are weighted and summed, a weight corresponding to the similarity S1 is greater than a weight corresponding to the similarity S2.
6. The method of claim 1, wherein, The final annotation results of the search result by the agents are synthesized to obtain the performance evaluation result of the search engine to be evaluated, comprising: synthesizing the final annotation results of the search result by the agents by voting or averaging to obtain a final relevance annotation result of the search result; performing statistics and analysis based on the final relevance annotation results of all the search results to obtain the performance evaluation result of the search engine to be evaluated.
7. A large language model agent-based search engine performance evaluation system, characterized in that, The system comprises: a data acquisition module configured to perform a search of a query set in a search engine to be evaluated to obtain a search result set; wherein the query set contains at least one query, and the search result set contains a search result of the query; a role definition module configured to define N roles of agents, wherein the agents are constructed based on a large language model, and the large language model has injected artificial relevance annotation experience into a prompt template thereof; a similarity calculation module configured to calculate a similarity between the query and the search result; and a performance evaluation module configured to synthesize final annotation results of the search result by the agents to obtain a performance evaluation result of the search engine to be evaluated. The retrieval result labeling module is configured to inject the role description of the agent, the query, the retrieval result, the similarity, and the historical labeling result of the retrieval result by the agent and the reason for the historical labeling result into the prompt template to obtain a current labeling result of the retrieval result by the agent and a reason for the current labeling result; after updating the historical labeling result and the reason for the historical labeling result based on the current labeling result and the reason for the current labeling result, the retrieval result labeling is re-executed, and after the number of re-executions reaches a set value, a final labeling result of the retrieval result by the agent is obtained; The evaluation result generation module is configured to comprehensively obtain the final labeling result of the retrieval result by each agent to obtain a performance evaluation result of the search engine to be evaluated.
8. An electronic device, comprising: The electronic device comprises a processor and a memory storing computer program instructions; and the processor implements the search engine performance evaluation method based on the large language model agent according to any one of claims 1-6 when executing the computer program instructions.
9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer program instructions, and the computer program instructions are executed by the processor to implement the search engine performance evaluation method based on the large language model agent according to any one of claims 1-6.
10. A computer program product, characterised in that, When the computer program product runs on the computer device, the computer device executes the search engine performance evaluation method based on the large language model agent according to any one of claims 1-6.