Method for optimizing search, electronic device, and program product for enhancing generation system

Abstract

This disclosure provides a method, electronic device, and program product for optimizing a retrieval enhancement generation system. The method includes: obtaining an original question set and an original customer identifier set; processing the original question set based on multiple large language models to obtain standard retrieval results; formulating multiple optimization strategies to be evaluated; processing the original question set based on the retrieval enhancement generation system and each optimization strategy to be evaluated to obtain retrieval results corresponding to each optimization strategy; evaluating the retrieval results corresponding to each optimization strategy based on the standard retrieval results to obtain evaluation results corresponding to each optimization strategy; and selecting a target optimization strategy from the multiple optimization strategies based on the multiple evaluation results.

Description

Technical Field

[0001] This disclosure relates to a method, electronic device, and program product for optimizing a search enhancement generation system. Background Technology

[0002] In the real estate brokerage industry, building accurate customer profiles is the core foundation for achieving efficient property matching and improving business conversion rates. While structured data is easy to query, unstructured data, such as conversation records containing customers' true intentions, is difficult to utilize effectively. To address these technical issues, vectorization techniques are used for processing. However, when faced with various retrieval optimization schemes, there is a lack of a scientific, quantifiable, and attributable evaluation method to verify the real improvement effect of different strategies on matching accuracy. This leads to technology selection relying on experience, high trial-and-error costs, and difficulty in systematically improving the intelligence level of accurate customer matching. Summary of the Invention

[0003] This disclosure provides a method, electronic device, and program product for optimizing and enhancing a search generation system.

[0004] According to one aspect of this disclosure, a method for optimizing a retrieval enhancement generation system is provided, comprising: The process involves: acquiring the original question set and the original customer identifier set; processing the original question set using multiple large language models to obtain standard search results; formulating multiple optimization strategies to be evaluated; processing the original question set using a search enhancement generation system and each optimization strategy to obtain search results corresponding to each strategy; evaluating the search results corresponding to each optimization strategy based on the standard search results to obtain evaluation results for each strategy; and selecting the target optimization strategy from the multiple optimization strategies based on the evaluation results.

[0005] Based on one aspect of the technical solution, standard search results were established, and multiple optimization strategies to be evaluated were formulated. The search results corresponding to each optimization strategy were compared with the standard search results to obtain the evaluation results for each strategy. The optimization effects of various optimization strategies on the search enhancement generation system were quantified using the evaluation results for each strategy, and the target optimization strategy was selected from the multiple strategies based on the optimization effects. This disclosed technical solution quantifies the optimization effects of various optimization strategies on the search enhancement generation system, thereby accurately selecting the optimization scheme for the search enhancement generation system. This, in turn, enables the adjusted search enhancement generation system to accurately recommend suitable customers to properties, reducing invalid recommendations and improving business conversion capabilities.

[0006] According to at least one embodiment of this disclosure, the original question set includes multiple original questions, and the original customer identifier set includes multiple original customer identifiers; the standard retrieval result includes multiple standard matching groups, each standard matching group including an original question and a list of original customer identifiers matching the original question; the list of original customer identifiers includes multiple original customer identifiers; the original question set is processed based on multiple large language models to obtain the standard retrieval result, including: for each large language model, processing each original question based on the large language model to obtain a list of original customer identifiers corresponding to each original question; for each original question, counting the number of matches between the original question and each corresponding original customer identifier; selecting multiple standard matching groups from multiple sets of corresponding original questions and original customer identifier lists based on multiple matching counts; the number of matches corresponding to the standard matching group is greater than the number of matches threshold.

[0007] According to at least one embodiment of this disclosure, the method further includes: processing the original question set based on the retrieval enhancement generation system to obtain initial retrieval results; the initial retrieval results include multiple initial matching groups, each initial matching group including the original question and a list of original customer identifiers corresponding to the original question; performing a differential analysis on the initial retrieval results and standard retrieval results to obtain an optimization scheme for the retrieval enhancement generation system; the optimization scheme includes multiple optimization algorithms.

[0008] According to at least one embodiment of this disclosure, the optimization strategy to be evaluated is a basic optimization strategy or a combined optimization strategy; multiple optimization strategies to be evaluated are formulated, including: selecting multiple optimization algorithms based on the optimization scheme of the retrieval enhancement generation system; selecting one optimization algorithm from multiple optimization algorithms to obtain a basic optimization strategy; and selecting at least two optimization algorithms from multiple optimization algorithms and determining the runtime order of at least two optimization algorithms to obtain a combined optimization strategy.

[0009] According to at least one embodiment of this disclosure, the basic optimization strategy includes an optimized semantic algorithm; the combined optimization strategy includes an optimized semantic algorithm, an optimized ranking algorithm, and an optimized vector algorithm; the original question set is processed based on the retrieval enhancement generation system and each optimization strategy to be evaluated to obtain the retrieval results to be evaluated corresponding to each optimization strategy, including: processing the original question set based on the retrieval enhancement generation system and the basic optimization strategy to obtain the retrieval results to be evaluated corresponding to the basic optimization strategy; and processing the original question set based on the retrieval enhancement generation system and the combined optimization strategy to obtain the retrieval results to be evaluated corresponding to the combined optimization strategy.

[0010] According to at least one embodiment of this disclosure, the method includes: for each combinatorial optimization strategy, deleting any one of a plurality of optimization algorithms to obtain an adjusted combinatorial optimization strategy; processing the original question set based on a retrieval enhancement generation system and the adjusted combinatorial optimization strategy to obtain adjusted retrieval results; the adjusted retrieval results include a plurality of adjusted matching groups, each adjusted matching group including the original question and an adjusted list of original customer identifiers corresponding to the original question; comparing each adjusted matching group with the corresponding standard matching group to obtain an adjusted comparison result for each adjusted matching group; statistically analyzing the plurality of adjusted comparison results to obtain an adjusted evaluation result corresponding to the adjusted combinatorial optimization strategy; and determining whether to retain the deleted optimization algorithm based on the adjusted evaluation result.

[0011] According to at least one embodiment of this disclosure, the method further includes: for each combinatorial optimization strategy, obtaining a replacement algorithm; replacing any one of the multiple optimization algorithms with the replacement algorithm to obtain an adjusted combinatorial optimization strategy; processing the original question set based on the retrieval enhancement generation system and the adjusted combinatorial optimization strategy to obtain adjusted retrieval results; the adjusted retrieval results include multiple adjusted matching groups, each adjusted matching group including the original question and the adjusted original customer identifier list corresponding to the original question; comparing each adjusted matching group with the corresponding standard matching group to obtain an adjusted comparison result for each adjusted matching group; statistically analyzing multiple adjusted comparison results to obtain an adjusted evaluation result corresponding to the adjusted combinatorial optimization strategy; and determining whether to retain the replacement algorithm based on the adjusted evaluation result.

[0012] According to at least one embodiment of this disclosure, the method further includes: for each combinatorial optimization strategy, obtaining an addition algorithm, adding the addition algorithm to the combinatorial optimization strategy to obtain an adjusted combinatorial optimization strategy; processing the original question set based on the retrieval enhancement generation system and the adjusted combinatorial optimization strategy to obtain adjusted retrieval results; the adjusted retrieval results include multiple adjusted matching groups, each adjusted matching group including the original question and the adjusted original customer identifier list corresponding to the original question; comparing each adjusted matching group with the corresponding standard matching group to obtain an adjusted comparison result for each adjusted matching group; statistically analyzing multiple adjusted comparison results to obtain an adjusted evaluation result corresponding to the adjusted combinatorial optimization strategy; and determining whether to retain the addition algorithm based on the adjusted evaluation result.

[0013] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform a method of an optimized retrieval enhancement generation system according to any embodiment of this disclosure.

[0014] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a method for optimizing a search-enhancing generation system according to any embodiment of this disclosure. Attached Figure Description

[0015] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0016] Figure 1 This is a schematic diagram of the overall process of an optimized retrieval enhancement generation system according to one embodiment of the present disclosure.

[0017] Figure 2 This is a flowchart illustrating step S102 of one embodiment of the present disclosure.

[0018] Figure 3 This is a flowchart illustrating the process of determining an optimization scheme in one embodiment of this disclosure.

[0019] Figure 4 This is a flowchart illustrating step S103 of one embodiment of the present disclosure.

[0020] Figure 5 This is a flowchart illustrating step S104 of one embodiment of the present disclosure.

[0021] Figure 6 This is a flowchart illustrating step S105 of one embodiment of the present disclosure.

[0022] Figure 7 This is a flowchart illustrating a method for quantifying an optimization algorithm according to the first embodiment of this disclosure.

[0023] Figure 8 This is a flowchart illustrating a method for quantifying an optimization algorithm according to the second embodiment of this disclosure.

[0024] Figure 9 This is a flowchart illustrating a method for quantifying an optimization algorithm according to the third embodiment of this disclosure.

[0025] Figure 10 This is a schematic structural block diagram of an apparatus for an optimized retrieval enhancement generation system according to one embodiment of the present disclosure.

[0026] Figure 11 This is a schematic structural block diagram of an electronic device according to one embodiment of the present disclosure. Detailed Implementation

[0027] The present disclosure will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0028] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] Currently, customer data is typically categorized into structured and unstructured data. While structured data is convenient for database storage and rule-based queries, its limited information dimensions make it difficult to fully reflect the complex needs of customers; for example, structured data includes a customer's budget range, desired apartment type, and preferred location. Meanwhile, unstructured data, which possesses significant commercial value, often remains dormant due to a lack of effective processing methods, making it difficult to utilize systematically. Examples of unstructured data include multi-round conversations between customers and agents, descriptions of inquiry emails, annotations during property browsing, and subjective evaluations of interior design styles.

[0030] To address the aforementioned technical challenges, related technologies have attempted to utilize vectorization techniques for storing and semantically retrieving unstructured data. However, in practical applications, retrieval effectiveness heavily relies on the rationality of query construction and the choice of retrieval strategy. These technologies often only achieve simple keyword matching or basic vector similarity retrieval, failing to effectively bridge the semantic gap. This results in coarse-grained customer profiles and often inaccurate matching results between properties and customers. Furthermore, in complex property matching scenarios, while various retrieval optimization solutions exist, such as query rewriting, hybrid retrieval (keywords + vectors), and Large Language Model (LLM) optimization, the industry currently lacks a scientific, quantifiable, and attributable evaluation method. Consequently, it is impossible to accurately verify the true improvement in customer search accuracy achieved by each optimization solution, leading to a high dependence on experience for technology selection and subsequent optimization directions, resulting in high trial-and-error costs.

[0031] To address this, this disclosure proposes the following technical solution. In this solution, an ideal structure is defined by constructing a standard structure to define the problem. Then, various optimization strategies to be evaluated are integrated one by one into the retrieval enhancement generation system. Performance evaluation is conducted on a unified test set, and the optimal optimization strategy is automatically selected based on the evaluation results. This technical solution allows for the plug-and-play and flexible combination of different optimization strategies. By comparing the standard retrieval results with the retrieval results corresponding to each optimization strategy, the effectiveness of different optimization strategies can be quantitatively evaluated, thereby selecting the optimal optimization strategy.

[0032] To facilitate description and make the technical solutions of this disclosure easier to understand, the terminology of this disclosure will be explained before describing the technical solutions of this disclosure.

[0033] 1. Retrieval Enhancement Generation System: Through three steps of "retrieval-enhancement-generation", it retrieves relevant information from external knowledge bases (such as patent databases and corporate private documents) before generating answers from the large language model, and outputs it as "context" to the large language model.

[0034] 2. Large Language Model: This is a deep learning model (usually based on the Transformer architecture) that compresses language knowledge into hundreds of millions of parameters by training on massive amounts of text data. Essentially, it's a probabilistic predictor that predicts the next word based on the preceding context.

[0035] Figure 1 This is a schematic diagram of the overall flow of an optimized retrieval enhancement generation system according to one embodiment of the present disclosure. Figure 1 The method shown includes steps S101 to S106. This method can be executed by electronic devices such as mobile phones and tablets.

[0036] In step S101, the original question set and the original customer identifier set are obtained.

[0037] The original question set includes multiple original questions, the original customer identifier set includes multiple original customer identifiers, and the original customer identifier list includes multiple original customer identifiers.

[0038] The "original question set" refers to a series of query commands simulating real home-buying needs. These commands are typically in natural language and include restrictions on property attributes, such as location, price, and unit type. The "original customer identifier set" is the set of customer entity indexes to be matched, with each original customer identifier representing a potential customer with specific purchasing power or intention. The core of this step is to build a standardized "demand-supply" testing environment, where the "questions" represent the filtering conditions on the property side or the search commands on the demand side, and the "identifier set" represents multiple customers to be filtered.

[0039] In some examples, within the real estate brokerage scenario, the "original question set" specifically manifests as typical home-buying needs extracted from historical search logs or commonly used broker phrases. For instance, vague customer needs are transformed into standardized query statements: "Two-bedroom apartment in XX district with a down payment of under 2 million." The "original customer identifier set" is a list of active home-buying identifiers (such as "Customer A," "Customer B," and "Customer C") extracted from a customer relationship management (CRM) system.

[0040] For example, after obtaining the original question set and the original customer identifier set, the original question set and the original customer identifier set are cleaned, deduplicated and classified, and it is ensured that the original customer identifier set covers user groups with different budgets and preferences, thereby providing an accurate and reliable data foundation for subsequent evaluation.

[0041] For example, 5,000 conversations between users and real estate agents are randomly selected from a real estate transaction database over the past year. Natural language processing technology is used to extract user questions, forming an "original question set." For instance, the original question might be "A two-bedroom apartment in District XX with a down payment of under 2 million." The original customer identifier list includes "Customer A," "Customer B," and "Customer C." This can be understood as "Customer A," "Customer B," and "Customer C" in the original customer identifier list potentially buying "a two-bedroom apartment in District XX with a down payment of under 2 million."

[0042] In step S102, the original question set is processed based on multiple large language models to obtain standard retrieval results.

[0043] The standard search results include multiple standard matching groups, each containing the original question and a list of original customer identifiers that match the original question. The number of original customer identifiers included in the list may vary depending on the large language model output. Standard search results can also be described as the "Gold Standard"; they can be used to evaluate recall, precision, validation re-ranking, and the effectiveness of optimization strategies.

[0044] In some examples, related technologies suffer from two major problems in obtaining standard retrieval results: high annotation costs and slow annotation processes. To address this issue, this disclosure utilizes multiple independent large language models to perform deep semantic understanding and logical reasoning on each original question in the "original question set" to obtain standard retrieval results. For example, the original question is "Two-bedroom apartment in XX district with a down payment of less than 2 million." The large language model, based on a broad knowledge base, infers and outputs the "ideal customer list" that best meets this requirement; the knowledge base includes common knowledge about housing prices, location, and customer purchasing power. By integrating the outputs of multiple large language models, some original customer identifiers with high confidence levels for this original question are identified as benchmarks for subsequent evaluation of the algorithm's performance.

[0045] For example, six or more large language models with different architectures are selected. The original question set is input into each large language model, and the output search results are the original customer identifier lists corresponding to each original question. This list of original customer identifiers represents the most likely customers to complete a transaction or meet the criteria for that original question. For each original question, if the original customer identifier lists output by multiple large language models are the same or overlap, then that original customer identifier list matches the original question; if multiple large language models output multiple original customer identifier lists, then the original customer identifiers with higher overlap are the original customer identifiers that match the original question. The final determined "standard search results" include multiple original customer identifiers with high confidence corresponding to each original question. For example, for an original question "Two-bedroom apartment in XX district with a down payment of less than 2 million," if the first three original customer identifier lists output by each large language model are "Customer A," "Customer B," and "Customer C," then the original question matches "Customer A," "Customer B," and "Customer C," which means the original question corresponds to "Customer A," "Customer B," and "Customer C." For example, given the original question "Two-bedroom apartment in XX district with a down payment of less than 2 million", six out of ten large language models output the original customer identifier list as "Customer A", "Customer B", and "Customer C"; two large language models output the original customer identifier list as "Customer B" and "Customer C"; and one large language model outputs the original customer identifier list as "Customer A" and "Customer C". Therefore, "Customer A" matches the original question.

[0046] For example, manual sampling and verification were also used for the above-mentioned standard search results to further ensure their reliability. For instance, 50% of the data in the standard search results were manually sampled and verified. For each set of original questions and corresponding original customer identifier lists in the 50% of the standard search results, if the verification result showed that the original question matched a portion of the original customer identifiers in the original customer identifier list, but did not match the remaining portion of the original customer identifiers in the original customer identifier list, then only the portion of the original customer identifiers in the original customer identifier list was retained, and the remaining portion of the original customer identifiers in the original customer identifier list was removed.

[0047] In step S103, various optimization strategies to be evaluated are formulated.

[0048] The different optimization strategies to be evaluated correspond to different processing stages in the "original question - original customer identifier list" matching process. These strategies aim to solve how to more accurately retrieve customers from the "original customer identifier set" that match the description of the "original question set". The various optimization strategies to be evaluated cover a variety of different optimization algorithms, including query understanding, hybrid retrieval, and re-ranking.

[0049] In step S104, the original problem set is processed based on the retrieval enhancement generation system and each optimization strategy to be evaluated, and the retrieval results corresponding to each optimization strategy to be evaluated are obtained.

[0050] In some examples, each optimization strategy to be evaluated is combined with a detection enhancement generation system to obtain an adjusted retrieval enhancement generation system corresponding to each optimization strategy to be evaluated; the original question set is then processed based on each adjusted retrieval enhancement generation system to obtain the retrieval results to be evaluated for each adjusted retrieval enhancement generation system.

[0051] In step S105, the evaluation results of each optimization strategy to be evaluated are evaluated based on the standard search results to obtain the evaluation results of each optimization strategy to be evaluated.

[0052] In some examples, the consistency between the search results for each evaluation strategy and the standard search results is compared. The degree of consistency is used to characterize the evaluation results, quantifying the optimization effect of each evaluation strategy on the search enhancement generation system. Specifically, if the consistency between the search results for each evaluation strategy and the standard search results is high, the optimization effect of that strategy on the search enhancement generation system is good; if the consistency between the search results for each evaluation strategy and the standard search results is low, the optimization effect of that strategy on the search enhancement generation system is poor.

[0053] In step S106, the target optimization strategy is selected from a variety of optimization strategies to be evaluated based on multiple evaluation results.

[0054] In some examples, the optimization strategies to be evaluated that have a good effect on the retrieval enhancement generation system are retained, i.e., the target optimization strategies; while the optimization strategies to be evaluated that have a poor effect on the retrieval enhancement generation system are removed.

[0055] For example, the optimization effects of various optimization strategies to be evaluated are sorted in descending order based on multiple evaluation results to obtain an effect list; the first optimization strategy to be evaluated in the effect list is taken as the target optimization strategy.

[0056] This disclosure provides a method for optimizing a search enhancement generation system. It sets standard search results and formulates multiple optimization strategies to be evaluated. The search results corresponding to each optimization strategy are compared with the standard search results to obtain the evaluation results for each strategy. The optimization effect of each optimization strategy on the search enhancement generation system is quantified using the evaluation results, and a target optimization strategy is selected from the multiple strategies based on the optimization effect. This technical solution quantifies the optimization effect of various optimization strategies on the search enhancement generation system, thereby accurately selecting the optimization scheme for the system. Ultimately, this enables accurate recommendation of suitable customers to properties based on the adjusted search enhancement generation system, reducing invalid recommendations and improving business conversion capabilities.

[0057] Figure 2 This is a flowchart illustrating step S102 of one embodiment of this disclosure. Figure 2 The method shown includes steps S1021 to S1023.

[0058] In step S1021, for each large language model, each original question is processed based on the large language model to obtain a list of original customer identifiers corresponding to each original question.

[0059] In some examples, this disclosure uses multiple different large language models to perform semantic reasoning and matching on each original question in the original question set. Specifically, multiple large language models are invoked in parallel to perform matching analysis on each original question with multiple original customer identifiers in the original customer identifier set, resulting in a list of original customer identifiers that match each original question. It is understood that each original question may match multiple original customer identifiers. For example, the original question set includes N original questions, the original customer identifier set includes K customers, and a total of M large language models output N×K×M sets (original questions, lists of original customer identifiers). Here, N, K, and M are all integers greater than or equal to 1.

[0060] In step S1022, for each original question, the number of times the original question matches each corresponding list of original customer identifiers is counted.

[0061] In some examples, the original customer identifiers corresponding to each original question output by all large language models are iterated through, and the number of matches for each "original question - original customer identifier" pair is counted. The higher the number of matches, the higher the consistency of the "original question - original customer identifier" pair across different large language models, and the higher its reliability as a standard matching pair. For example, if the first large language model outputs "Customer A", "Customer B", "Customer C", "Customer D"; the second large language model outputs "Customer B", "Customer C", "Customer D"; and the third large language model outputs "Customer A", "Customer C", "Customer E", then "Customer A" has 2 matches, "Customer B" has 2 matches, "Customer C" has 3 matches, "Customer D" has 2 matches, and "Customer E" has 1 match. Therefore, "Original Question - Customer C" has the highest consistency across different large language models, and its reliability as a standard matching pair is also the highest.

[0062] In step S1023, multiple standard matching groups are selected from multiple sets of corresponding original questions and original customer identifier lists based on multiple matching counts.

[0063] Among them, the number of matches corresponding to the standard matching group is greater than the number of matches threshold; each original question can correspond to multiple standard matching groups.

[0064] In some examples, a matching threshold is pre-set. Only when the number of matches for each set of original questions and original customer identifiers exceeds the threshold is that set of original questions and original customer identifiers considered as a standard matching group. For example, if the matching threshold is M×50%, then in step S1022, the matching counts for (original question, customer A), (original question, customer B), (original question, customer C), and (original question, customer D) are greater than 3×50%. Therefore, the original customer identifier list corresponding to this original question, including "customer A", "customer B", "customer C", and "customer D", constitutes a standard matching group. The matching count for "customer E" is less than 3×50%, therefore, the original question does not match "customer E".

[0065] The above technical solution processes the original question set using multiple large language models to obtain a list of original customer identifiers corresponding to each original question output by each large language model. Multiple standard matching groups are then selected from these groups based on the number of matches between each pair (original question, original customer identifier). This ensures the accuracy of the standard matching groups and avoids evaluation bias caused by errors in the standard matching groups.

[0066] Figure 3 This is a flowchart illustrating the determination of an optimization scheme in one embodiment of this disclosure. For example... Figure 3The method shown includes steps S301 to S302.

[0067] In step S301, the original question set is processed based on the retrieval enhancement generation system to obtain initial retrieval results.

[0068] The initial search results include multiple initial matching groups, each of which includes the original question and a list of original customer identifiers corresponding to the original question.

[0069] In some examples, the original question set is input into an unoptimized retrieval enhancement generation system, triggering it to execute a complete retrieval and generation process. The result is the initial retrieval results returned by the retrieval enhancement generation system based on its existing algorithms, index structure, and model parameters (original question, original customer identifier list). These initial retrieval results represent the basic performance of the retrieval enhancement generation system before this round of optimization.

[0070] For example, for each original question, the retrieval enhancement generation system segments and vectorizes the original question, performs an approximate nearest neighbor search in the customer profile vector library, and outputs a list of original customer identifiers corresponding to the original question. The customer profile vector library is prepared in advance based on historical customer data.

[0071] In step S302, a differential analysis is performed on the initial search results and the standard search results to obtain an optimized scheme for the search enhancement generation system.

[0072] The optimization scheme includes a variety of optimization algorithms.

[0073] In some examples, in order to develop meaningful optimization strategies and reduce meaningless evaluations, this disclosure performs a differential analysis on the initial search results and standard search results before developing optimization strategies to identify problems in the search enhancement generation system and preliminarily determine optimization schemes based on these problems.

[0074] For example, a multi-dimensional comparative analysis is performed on the initial search results and the standard search results to identify specific problems in the search enhancement generation system. These specific problems include semantic gap, poor ranking quality, and vector drift. Semantic gap can be understood as a mismatch between keyword matching and the user's true intent. Poor ranking quality can be understood as a failure to align the retrieved data with intent. Vector drift can be understood as a lack of fine-tuning of the large model within the real estate vertical, resulting in a misunderstanding of professional terms such as "school district" and "five-year ownership and only property."

[0075] For example, this disclosure identifies optimization schemes to address specific problems in the retrieval enhancement generation system. For instance, to address the "semantic gap," an optimized semantic algorithm is selected for the retrieval recall step. This algorithm converts text into vectors, allowing content with different expressions but the same intent to be matched, thus solving the problem of not understanding user intent. The optimized semantic algorithm is either the k-Nearest Neighbors (KNN) algorithm or a semantic recall algorithm. To address the "poor ranking quality," an optimized reordering algorithm is selected for the reordering step. This algorithm performs secondary screening and reordering of the initial results, preventing structures with higher matching degrees from being ranked after those with lower matching degrees. The optimization algorithm for reordering is based on a large language model. To address the "vector drift," an optimized vector algorithm is selected for the retrieval recall step. This algorithm constrains the retrieval range, preventing retrieval results from deviating from the expected range and solving the problem of inaccurate vector retrieval. For example, the optimization algorithms for retrieval recall include the Best Match 25 algorithm (BM25+KNN) and the Reciprocal Rank Fusion (RRF) algorithm.

[0076] Figure 4 This is a flowchart illustrating step S103 of one embodiment of this disclosure. Figure 4 The method shown includes steps S1031 to S1032.

[0077] In step S1031, multiple optimization algorithms are selected based on the optimization scheme of the retrieval enhancement generation system.

[0078] Among them, various optimization algorithms include, but are not limited to, semantic optimization algorithms, rearrangement optimization algorithms, and vector optimization algorithms.

[0079] In step S1032, an optimization algorithm is selected from multiple optimization algorithms to obtain a basic optimization strategy; and at least two optimization algorithms are selected from multiple optimization algorithms, and the execution order of the at least two optimization algorithms is determined to obtain a combined optimization strategy.

[0080] Among them, the optimization strategies to be evaluated are basic optimization strategies or combined optimization strategies.

[0081] In some examples, the optimization algorithms to be evaluated are divided into basic optimization strategies and combined optimization strategies because a basic optimization strategy includes only one optimization algorithm. Therefore, the evaluation results corresponding to this basic optimization strategy can quantify the optimization effect of that algorithm on the retrieval augmentation generation system. For example, a basic optimization strategy includes any one of the following: semantic optimization algorithm, ranking optimization algorithm, and vector optimization algorithm. By evaluating the combined optimization strategy, the optimization effect of at least two combined optimization algorithms on the retrieval augmentation generation system can be quantified. For example, a combined optimization strategy includes any two of the following: semantic optimization algorithm, ranking optimization algorithm, and vector optimization algorithm. Another example is a combined optimization strategy including semantic optimization algorithm, ranking optimization algorithm, and vector optimization algorithm. For instance, the first basic optimization strategy is KNN. The second basic optimization strategy is semantic recall. The first combined optimization strategy is KNN and LLM. The second combined optimization strategy is BM25 and RRF. The third combined optimization strategy is BM25, RRF, and LLM. The fourth optimization strategy is semantic recall and LLM.

[0082] It should be noted that for the combined optimization strategy, the runtime sequence also needs to be determined based on the actual execution steps of the retrieval enhancement generation system corresponding to various algorithms during operation, so as to ensure that the adjusted retrieval enhancement generation system can operate normally.

[0083] Figure 5 This is a flowchart illustrating step S104 of one embodiment of this disclosure. Figure 5 The method shown includes steps S1041 to S1042.

[0084] In step S1041, the original problem set is processed based on the retrieval enhancement generation system and the basic optimization strategy to obtain the retrieval results to be evaluated corresponding to the basic optimization strategy.

[0085] In step S1042, the original problem set is processed based on the retrieval enhancement generation system and the combinatorial optimization strategy to obtain the retrieval results to be evaluated corresponding to the combinatorial optimization strategy.

[0086] Figure 6 This is a flowchart illustrating step S105 of one embodiment of this disclosure. Figure 6 The method shown includes steps S1051 to S1052.

[0087] In step S1051, for each optimization strategy to be evaluated, the retrieval results are compared between each matching group to be evaluated and the corresponding standard matching group to obtain the comparison results of each matching group to be evaluated.

[0088] In some examples, to ensure the accuracy of the evaluation results, each matching group to be evaluated is compared with the corresponding standard matching group based on multiple indicators.

[0089] For example, Recall@K is calculated for each original question. Recall@K is used to confirm whether the correct original customer identifiers have been found.

[0090] For example, Recall@K = (the number of matches among the first K original customer identifiers in the original customer identifier list of the match group to be evaluated) / (the number of original identifiers in the original customer identifier list of the standard match group). Here, the number of matches among the first K original customer identifiers in the original customer identifier list of the match group to be evaluated can be understood as the number of matches between the first K original customer identifiers in the original customer identifier list of the match group and the original identifiers in the original customer identifier list of the standard match group.

[0091] When the number of original identifiers in the original customer identifier list of the standard matching group is 10, and the number of matches among the top K original customer identifiers in the original customer identifier list of the evaluation matching group is 5, then Recall@K is 50%. The average Recall@K (recall rate) of all original questions is calculated, and the optimization effect of the corresponding optimization strategy is evaluated using this average. For example, an increase in Recall@K indicates that the optimization strategy under evaluation helps the retrieval enhancement generation system uncover potential customers that were previously not matched by keywords.

[0092] For example, the Precision@K is calculated for each original question. Precision@K is used to identify redundant information. For example, Precision@K = (the number of matches in the first K original customer identifiers of the original customer identifier list in the match group to be evaluated) / K.

[0093] For example, the MRR (Mean Reverse Rank) is calculated. The MRR is used to determine the position of the first correctly identified original customer identifier in the list of original customer identifiers; the earlier the position, the higher the score.

[0094] For example, nDCG@K (normalized loss cumulative gain) is calculated. nDCG@K is used to confirm whether the correct original customer identifier is in the correct position in the original customer identifier list.

[0095] In step S1052, multiple comparison results are statistically analyzed to obtain the evaluation results corresponding to the optimization strategy to be evaluated.

[0096] The search results to be evaluated include multiple matching groups to be evaluated. Each matching group to be evaluated includes the original question and a list of original customer identifiers corresponding to the original question.

[0097] Based on the comparison results of multiple indicators in step S1051, the optimization effect of each optimization strategy to be evaluated on the retrieval enhancement generation system is assessed.

[0098] Figure 7 This is a flowchart illustrating a method for quantifying an optimization algorithm according to the first embodiment of this disclosure. Figure 7 The method shown includes steps S701 to S705.

[0099] In step S701, for each combined optimization strategy, any one of the multiple optimization algorithms is deleted to obtain the adjusted combined optimization strategy.

[0100] For each combination optimization strategy, its optimization effect on the retrieval enhancement generation system may come from one or more optimization algorithms. To further confirm whether it is one or more optimization algorithms that have achieved the optimization effect, it is necessary to delete any one optimization algorithm to obtain the adjusted combination optimization strategy.

[0101] In step S702, the original question set is processed based on the retrieval enhancement generation system and the adjusted combined optimization strategy to obtain the adjusted retrieval results.

[0102] The adjusted search results include multiple adjusted matching groups, each of which includes the original question and the corresponding adjusted list of original customer identifiers.

[0103] In step S703, each adjusted matching group is compared with the corresponding standard matching group to obtain the adjusted comparison result of each adjusted matching group.

[0104] In step S704, multiple adjusted comparison results are statistically analyzed to obtain the adjusted evaluation results corresponding to the adjusted combined optimization strategy.

[0105] It should be noted that steps S703 and S704 are the same as steps S1051 and S1052 above, and will not be described again.

[0106] In step S705, it is determined whether to retain the deleted optimization algorithm based on the adjusted evaluation results.

[0107] Specifically, if the adjusted evaluation results show that the optimization effect of the adjusted combined optimization strategy deteriorates, it indicates that the deleted optimization algorithm has a certain optimization effect and needs to be retained. If the adjusted evaluation results show that the optimization effect of the adjusted combined optimization strategy does not deteriorate, it indicates that the remaining optimization algorithms have a good optimization effect, and the deleted optimization algorithm has no optimization effect and does not need to be retained. This achieves the quantification of the optimization effect of the deleted optimization algorithm. In addition, the optimization effect ranking of different adjusted combined optimization strategies can be determined based on different adjusted evaluation results, so as to better select the target optimization strategy.

[0108] In addition, the optimization effect of the adjusted combinatorial optimization strategy can be compared with that of the corresponding basic optimization strategy by evaluating the results. This will determine the optimization effect of the adjusted combinatorial optimization strategy after removing the optimization algorithms in the basic optimization strategy, and thus determine whether to continue to remove useless optimization algorithms from the remaining optimization strategies.

[0109] Figure 8 This is a flowchart illustrating a method for quantifying an optimization algorithm according to a second embodiment of this disclosure. Figure 8 The method shown includes steps S801 to S805.

[0110] In step S801, for each combined optimization strategy, a replacement algorithm is obtained, and any one of the multiple optimization algorithms is replaced based on the replacement algorithm to obtain the adjusted combined optimization strategy.

[0111] To determine whether the optimization effect of each combined optimization strategy on the retrieval enhancement generation system can be further improved, it is necessary to replace any one of the optimization algorithms to obtain the adjusted combined optimization strategy.

[0112] In step S802, the original question set is processed based on the retrieval enhancement generation system and the adjusted combined optimization strategy to obtain the adjusted retrieval results.

[0113] The adjusted search results include multiple adjusted matching groups, each of which includes the original question and the corresponding adjusted list of original customer identifiers.

[0114] In step S803, each adjusted matching group is compared with the corresponding standard matching group to obtain the adjusted comparison result of each adjusted matching group.

[0115] In step S804, multiple adjusted comparison results are statistically analyzed to obtain the adjusted evaluation results corresponding to the adjusted combined optimization strategy.

[0116] In step S805, it is determined whether to retain the replacement algorithm based on the adjusted evaluation results.

[0117] The replacement algorithm is any one of many optimization algorithms.

[0118] It should be noted that steps S802 to S805 are the same as steps S702 to S705 above, and will not be described again.

[0119] Figure 9 This is a flowchart illustrating a method for quantifying an optimization algorithm according to the third embodiment of this disclosure. Figure 9 The method shown includes steps S901 to S905.

[0120] In step S901, for each combinatorial optimization strategy, an additional algorithm is obtained and added to the combinatorial optimization strategy to obtain the adjusted combinatorial optimization strategy.

[0121] To determine whether the optimization effect of each combined optimization strategy on the retrieval enhancement generation system can be further improved, it is necessary to add any optimization algorithm to obtain the adjusted combined optimization strategy.

[0122] In step S902, the original question set is processed based on the retrieval enhancement generation system and the adjusted combined optimization strategy to obtain the adjusted retrieval results.

[0123] The adjusted search results include multiple adjusted matching groups, each of which includes the original question and the corresponding adjusted list of original customer identifiers.

[0124] In step S903, each adjusted matching group is compared with the corresponding standard matching group to obtain the adjusted comparison result of each adjusted matching group.

[0125] In step S904, multiple adjusted comparison results are statistically analyzed to obtain the adjusted evaluation results corresponding to the adjusted combined optimization strategy.

[0126] In step S905, it is determined whether to retain the added algorithm based on the adjusted evaluation results.

[0127] Among them, the addition algorithm is any one of a variety of optimization algorithms.

[0128] It should be noted that steps S902 to S905 are the same as steps S702 to S705 above, and will not be described again.

[0129] The aforementioned technical solution, by deleting, replacing, or adding optimization algorithms, yields an adjusted combined optimization strategy. By evaluating the optimization effect of this adjusted combined optimization strategy, the effectiveness of the deleted, replaced, or added algorithms can be quantified, thus allowing for a more accurate determination of the final combined optimization strategy. This disclosure successfully applies the ablation experiment controlled variable method—a scientific research paradigm—to the precise customer matching scenario in the vertical field of real estate brokerage. It solves the industry challenge of quantifying and evaluating the contribution of a single component in a complex Artificial Intelligence (AI) system, enabling the selection of the most effective target optimization strategy for a specific type of customer query. This provides precise guidance for R&D resource allocation, significantly reducing trial-and-error costs and improving optimization efficiency.

[0130] Furthermore, this disclosure also provides a plug-and-play evaluation system architecture that allows for flexible combination of any optimization algorithm. This architecture can not only be used for evaluation but also support rapid iteration and verification of various optimization algorithms.

[0131] This disclosure also provides an apparatus for optimizing and enhancing a search generation system. Figure 10 This is a schematic structural block diagram of an apparatus for an optimized retrieval enhancement generation system according to one embodiment of this disclosure. Figure 10 As shown, the device includes: Module 1001 is used to obtain the original question set and the original customer identifier set; Processing module 1002 is used to process the original question set based on multiple large language models to obtain standard search results; Processing module 1002 is used to formulate various optimization strategies to be evaluated; Processing module 1002 is used to process the original question set based on the retrieval enhancement generation system and each optimization strategy to be evaluated, and to obtain the retrieval results to be evaluated corresponding to each optimization strategy to be evaluated; Processing module 1002 is used to evaluate the retrieval results corresponding to each optimization strategy to be evaluated based on the standard retrieval results, and obtain the evaluation results corresponding to each optimization strategy to be evaluated. The processing module 1002 is used to select the target optimization strategy from a variety of optimization strategies to be evaluated based on multiple evaluation results.

[0132] The specific implementation process of the functions and roles of each module in the above-mentioned device is detailed in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here. The execution subject of the method for optimizing the retrieval enhancement generation system in the specific embodiments of this disclosure can be an electronic device such as a server (including a local server or a cloud server).

[0133] Therefore, based on any of the above embodiments, this disclosure also provides an electronic device that can execute the method of the optimized retrieval enhancement generation system of any of the embodiments described above.

[0134] Figure 11 This is a schematic structural block diagram of an electronic device according to one embodiment of the present disclosure.

[0135] The hardware architecture of electronic devices / devices can be implemented using a bus architecture. A bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripherals, voltage regulators, power management circuits, external antennas, etc. Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one connection line is used in this diagram, but this does not indicate that there is only one bus or one type of bus.

[0136] For ease of explanation, certain steps of the above method are described in relation to modules. It should be understood that the corresponding module performing one or more steps of the above method may be one or more hardware modules specifically configured to perform the corresponding step, or implemented by a processor configured to perform the corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination thereof.

[0137] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0138] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0139] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0140] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0142] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0143] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0144] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A method of optimizing a search enhancement generation system, the method comprising: include: Obtain the original question set and the original customer identifier set; The original question set is processed using multiple large language models to obtain standard search results; Develop multiple optimization strategies to be evaluated; The original question set is processed based on the retrieval enhancement generation system and each of the aforementioned optimization strategies to be evaluated, to obtain the retrieval results to be evaluated corresponding to each of the aforementioned optimization strategies; Based on the standard search results, the search results corresponding to each optimization strategy to be evaluated are evaluated to obtain the evaluation results corresponding to each optimization strategy to be evaluated. Based on multiple evaluation results, a target optimization strategy is selected from a variety of optimization strategies to be evaluated.

2. The method of claim 1, wherein, The original question set includes multiple original questions, and the original customer identifier set includes multiple original customer identifiers; the standard search results include multiple standard matching groups, each of the standard matching groups includes the original question and a list of original customer identifiers that match the original question; the list of original customer identifiers includes multiple original customer identifiers; The original question set is processed using multiple large language models to obtain standard search results, including: For each of the large language models, each of the original questions is processed based on the large language model to obtain the list of original customer identifiers corresponding to each of the original questions; For each of the original questions, count the number of times the original question matches each of the corresponding original customer identifiers; Multiple standard matching groups are selected from multiple sets of corresponding original questions and original customer identifier lists based on multiple matching counts; the matching counts corresponding to the standard matching groups are greater than the matching count threshold.

3. The method as described in claim 1, characterized in that, The method further includes: The original question set is processed based on the retrieval enhancement generation system to obtain initial retrieval results; The initial search results include multiple initial matching groups, and each initial matching group includes the original question and the list of original customer identifiers corresponding to the original question; A differential analysis is performed on the initial search results and the standard search results to obtain an optimization scheme for the search enhancement generation system; the optimization scheme includes multiple optimization algorithms.

4. The method as described in claim 3, characterized in that, The optimization strategy to be evaluated is either a basic optimization strategy or a combined optimization strategy. Develop multiple optimization strategies to be evaluated, including: Based on the optimization scheme of the retrieval enhancement generation system, multiple optimization algorithms are selected; From the plurality of optimization algorithms, one optimization algorithm is selected to obtain a basic optimization strategy; and from the plurality of optimization algorithms, at least two optimization algorithms are selected, and the execution order of the at least two optimization algorithms is determined to obtain a combined optimization strategy.

5. The method as described in claim 4, characterized in that, The basic optimization strategy includes an optimized semantic algorithm; the combined optimization strategy includes the optimized semantic algorithm, the optimized sorting algorithm, and the optimized vector algorithm. Based on the retrieval enhancement generation system and each of the aforementioned optimization strategies to be evaluated, the original question set is processed to obtain the retrieval results to be evaluated corresponding to each of the aforementioned optimization strategies, including: Based on the retrieval enhancement generation system and the basic optimization strategy, the original question set is processed to obtain the retrieval result to be evaluated corresponding to the basic optimization strategy; The original question set is processed based on the retrieval enhancement generation system and the combined optimization strategy to obtain the retrieval results to be evaluated corresponding to the combined optimization strategy.

6. The method as described in claim 4, characterized in that, The method further includes: For each of the aforementioned combined optimization strategies, any one of the multiple optimization algorithms is deleted to obtain the adjusted combined optimization strategy; The original question set is processed based on the retrieval enhancement generation system and the adjusted combined optimization strategy to obtain the adjusted retrieval results; the adjusted retrieval results include multiple adjusted matching groups, and each adjusted matching group includes the original question and the adjusted original customer identifier list corresponding to the original question; By comparing each adjusted matching group with the corresponding standard matching group, the adjusted comparison result of each adjusted matching group is obtained; By statistically analyzing multiple comparison results after the adjustments, the adjusted evaluation results corresponding to the adjusted combined optimization strategy are obtained. Based on the adjusted evaluation results, determine whether to retain the deleted optimization algorithm.

7. The method as described in claim 4, characterized in that, The method further includes: For each of the combined optimization strategies, a replacement algorithm is obtained, and any one of the multiple optimization algorithms is replaced based on the replacement algorithm to obtain the adjusted combined optimization strategy; The original question set is processed based on the retrieval enhancement generation system and the adjusted combined optimization strategy to obtain the adjusted retrieval results; the adjusted retrieval results include multiple adjusted matching groups, and each adjusted matching group includes the original question and the adjusted original customer identifier list corresponding to the original question; By comparing each adjusted matching group with the corresponding standard matching group, the adjusted comparison result of each adjusted matching group is obtained; By statistically analyzing multiple comparison results after the adjustments, the adjusted evaluation results corresponding to the adjusted combined optimization strategy are obtained. Based on the adjusted evaluation results, determine whether to retain the replacement algorithm.

8. The method as described in claim 4, characterized in that, The method further includes: For each of the aforementioned combined optimization strategies, an addition algorithm is obtained, and the addition algorithm is added to the combined optimization strategy to obtain the adjusted combined optimization strategy; The original question set is processed based on the retrieval enhancement generation system and the adjusted combined optimization strategy to obtain the adjusted retrieval results; the adjusted retrieval results include multiple adjusted matching groups, and each adjusted matching group includes the original question and the adjusted original customer identifier list corresponding to the original question; By comparing each adjusted matching group with the corresponding standard matching group, the adjusted comparison result of each adjusted matching group is obtained; By statistically analyzing multiple comparison results after the adjustments, the adjusted evaluation results corresponding to the adjusted combined optimization strategy are obtained. Based on the adjusted evaluation results, determine whether to retain the added algorithm.

9. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the method of the optimized retrieval enhancement generation system according to any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of the optimized retrieval enhancement generation system as described in any one of claims 1 to 8.

Images