Model evaluation and replacement method and electronic device

Abstract

The application relates to the computer technical field and discloses a model evaluation and replacement method and an electronic device. The method comprises the following steps: obtaining a target knowledge base suitable for a preset service scenario; obtaining a test data set required for model evaluation based on historical interaction data and the target knowledge base; determining a plurality of to-be-evaluated models; performing performance evaluation on the plurality of to-be-evaluated models based on the target knowledge base and the test data set, analyzing the recall accuracy, semantic correlation and overall performance of each to-be-evaluated model to obtain performance evaluation results of each to-be-evaluated model; determining a to-be-replaced model according to the performance evaluation results of the to-be-evaluated models; and updating an original model to the to-be-replaced model. Through a systematic and automatic evaluation process and a replacement mechanism, the application improves the efficiency of model evaluation and updating and ensures that the system can maintain efficient and accurate performance in a variable business environment.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to a method for evaluating and replacing a model, and an electronic device. Background Technology

[0002] In intelligent customer service systems, the intent recognition module is a key component for understanding user needs, and its performance directly impacts the overall system performance. Currently, intelligent customer service systems rely on semantic representation models (such as embedding models) to identify user intent. With the rapid development of natural language processing, semantic representation model technology is constantly iterating, resulting in a new generation of more efficient and accurate models. These new-generation models typically possess stronger semantic representation capabilities and perform exceptionally well in similarity calculation and intent matching accuracy. Introducing these advanced semantic representation models into existing intelligent customer service systems presents a challenge: how to ensure that the actual performance of the new models outperforms the current models.

[0003] Therefore, relevant technologies evaluate new models to determine their effectiveness and decide whether to replace the old model. However, currently, the evaluation of new models is usually done manually, which is cumbersome and lacks a systematic approach. Summary of the Invention

[0004] One objective of this application is to provide a model evaluation and replacement method and electronic device to solve the technical problem of how to efficiently and systematically evaluate model performance, thereby enabling efficient and convenient verification of whether a new model is superior to the current model, and to replace it.

[0005] To address the aforementioned technical problems, one technical solution adopted in this application is: providing a model evaluation and replacement method, wherein the model is applied to a preset service scenario, the method comprising: acquiring a target knowledge base applicable to the preset service scenario; acquiring a test dataset required for model evaluation based on historical interaction data and the target knowledge base; the historical interaction data including real user questions, which are questions actually input by users and the system during historical interactions; determining multiple models to be evaluated; performing performance evaluation on the multiple models to be evaluated based on the target knowledge base and the test dataset, analyzing the recall accuracy, semantic relevance, and overall performance of each model to be evaluated, to obtain a performance evaluation result for each model to be evaluated; the recall accuracy is used to measure the matching degree between the information retrieved by the model to be evaluated and the standard questions in the target knowledge base, and the semantic relevance is used to measure the matching degree between the results generated by the model to be evaluated and the semantics of the user questions; determining a replacement model based on the performance evaluation results of the models to be evaluated; and updating the original model to the replacement model.

[0006] This method first acquires a target knowledge base applicable to a pre-defined service scenario, providing the core data source for model evaluation. The target knowledge base provides background knowledge for the model, ensuring the relevance and accuracy of the evaluation results. A test dataset is obtained based on historical interaction data and the target knowledge base. The design of the test dataset closely matches actual business scenarios and reflects the diversity of real user questions. Performance evaluation employs a multi-dimensional approach, including: recall accuracy (whether the model can accurately find knowledge items relevant to user questions); semantic relevance (whether the content of the recalled results is semantically consistent with the user's question); and overall performance. This multi-dimensional analysis method ensures the scientific rigor and accuracy of the model evaluation results. Finally, the original model is updated to the replacement model. This process can be automated, efficiently and seamlessly replacing poorly performing models. Therefore, this method, through a systematic and automated evaluation process and replacement mechanism, improves the efficiency of model evaluation and updating, ensuring that the system maintains efficient and accurate performance in a dynamic business environment.

[0007] Optionally, obtaining the test dataset required for model evaluation based on historical interaction data and the target knowledge base includes: obtaining real user questions based on historical interaction data; converting the real user questions into semantic vectors, and searching for entries related to the real user questions in the target knowledge base according to the semantic vectors, with the entries serving as recall results; generating system response results corresponding to the real user questions based on the target knowledge base and the historical interaction data; wherein, the real user questions, the recall results, and the system response results constitute the test dataset. By using real user questions and historical data, the test dataset can more realistically reflect user needs and improve the effectiveness of the evaluation.

[0008] Optionally, the real user queries include general questions, ambiguous questions, and unknown questions. By including different types of questions, the test dataset ensures that it reflects the diverse needs of users, helping the model respond effectively in various scenarios. Furthermore, by handling ambiguous and unknown questions, the model can learn how to handle complex and uncertain user queries, thereby enhancing its robustness and adaptability.

[0009] Optionally, the step of evaluating the performance of the multiple models to be evaluated based on the target knowledge base and the test dataset, and analyzing the recall accuracy, semantic relevance, and overall performance of each model to obtain the performance evaluation result of each model to be evaluated, includes: for any model to be evaluated, performing semantic vectorization processing on the knowledge entries in the target knowledge base using the model to generate a semantic vector for each knowledge entry, and storing the generated semantic vector in a vector database; generating a recall result list corresponding to the model to be evaluated based on the vector database; comparing the recall questions in the recall result list with the standard questions in the target knowledge base to evaluate the recall accuracy of the model to be evaluated; judging the semantic relevance of the recall questions in the recall result list with the standard questions in the target knowledge base to evaluate the semantic relevance of the model to be evaluated; and comprehensively evaluating the model to be evaluated based on the evaluation results corresponding to the recall accuracy and the evaluation results corresponding to the semantic relevance of the model to be evaluated to obtain the performance evaluation result of the model to be evaluated. By analyzing recall accuracy and semantic relevance, this method provides a comprehensive understanding of the model's performance across different dimensions, making the evaluation more scientific and accurate. Furthermore, its systematic approach ensures comprehensive analysis and optimization of the model, providing solid data support for subsequent decision-making and improvement.

[0010] Optionally, generating the recall result list corresponding to the model to be evaluated based on the vector database includes: semantically vectorizing real user questions in the test dataset using the model to be evaluated to generate a semantic vector for each real user question; comparing the similarity between the semantic vector of each real user question and the semantic vector of a knowledge item in the vector database to obtain knowledge items similar to the real user questions; and generating the recall result list corresponding to the model to be evaluated based on the obtained knowledge items similar to the real user questions. Here, semantic vectorization and similarity comparison ensure the relevance of the recall results to the user questions, improving the accuracy of the recall, which lays the foundation for subsequent model evaluation and optimization.

[0011] Optionally, the step of semantically relevance assessment of the recalled questions in the recall result list and the standard questions in the target knowledge base to evaluate the semantic relevance of the model to be evaluated includes: semantically comparing the recalled questions in the recall result list with the standard questions in the target knowledge base to determine whether the semantics of the recalled questions and the standard questions are consistent or similar; and assessing whether the recalled results in the recall result list are related to the standard questions in the target knowledge base. Through semantic comparison and relevance assessment, the performance of the model can be evaluated more accurately, ensuring that only relevant recalled results are considered, which also lays the foundation for subsequent model evaluation and optimization.

[0012] Optionally, the step of comprehensively evaluating the model to be evaluated based on the evaluation results corresponding to the recall accuracy and the semantic relevance of the model to be evaluated to obtain the performance evaluation result of the model to be evaluated includes: obtaining an accuracy score for the model to be evaluated for each question category based on the question categories corresponding to real user questions in the test dataset and the evaluation results corresponding to the recall accuracy of the model to be evaluated; obtaining a recall score for the model to be evaluated for each question category based on the question categories corresponding to real user questions in the test dataset and the evaluation results corresponding to the semantic relevance of the model to be evaluated; obtaining the weights corresponding to the question categories; and calculating the comprehensive score of the model to be evaluated based on the weights, the accuracy score, and the recall score to obtain the performance evaluation result of the model to be evaluated. This method explains how to calculate the comprehensive score. This process breaks down the evaluation results into each question category, enabling a deeper analysis of the model's performance on different types of questions and revealing the model's strengths and weaknesses. Considering both accuracy and recall simultaneously ensures that the model can not only accurately answer user questions but also cover more relevant questions, thereby improving the overall service quality. Finally, by introducing weights for each problem category, the impact of each category on the overall score can be adjusted based on actual business needs and user importance, ensuring that the evaluation results are more realistic. This method effectively improves the accuracy and practicality of the performance evaluation results of the model being evaluated.

[0013] Optionally, before performing the step of updating the original model to the model to be replaced, the method further includes: fitting a similarity distribution between the model to be replaced and the original model based on distribution alignment; performing distribution alignment between the model to be replaced and the original model according to the similarity distribution between the model to be replaced and the original model; verifying the alignment effect of the model to be replaced after distribution alignment; and if the alignment effect meets a preset condition, then performing the step of updating the original model to the model to be replaced. Here, distribution alignment ensures that the output of the model to be replaced is consistent with the original model, thereby avoiding performance degradation caused by model replacement.

[0014] Optionally, updating the original model to the model to be replaced includes: performing a vector update on the target knowledge base using the model to be replaced to generate an updated target knowledge base; obtaining the result generated by the model to be replaced when a first preset number of users send access requests to the updated target knowledge base, the result including a request error rate; if the request error rate is within a first preset range, then increasing the number of users accessing the updated target knowledge base based on the first preset number, and continuously monitoring the request error rate until a preset condition is met, and then completing the release of the model to be replaced, wherein the preset condition includes a request error rate within the first preset range and a number of users within a second preset range. By gradually expanding the user access scope, the risks that may arise from the launch of a new model are reduced, ensuring system stability; moreover, the entire model replacement process follows established procedures and standards, ensuring the systematic nature and consistency of the replacement process.

[0015] To address the aforementioned technical problems, one technical solution adopted in this application is to provide an electronic device, including a memory and a processor. The memory is connected to the processor, and the processor is configured to execute one or more computer programs stored in the memory. When executing the one or more computer programs, the processor enables the electronic device to implement an evaluation and replacement method for a model applied to the electronic device. This electronic device possesses the beneficial effects corresponding to the aforementioned evaluation and replacement method for a model applied to the electronic device. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1This is a flowchart of a model evaluation and replacement method provided in an embodiment of this application;

[0018] Figure 2 This is a flowchart of a method for obtaining the performance evaluation results of each model to be evaluated, provided in an embodiment of this application.

[0019] Figure 3 This is a flowchart of a method provided in this application for obtaining the performance evaluation result of a model to be evaluated based on the evaluation result corresponding to the recall accuracy of the model to be evaluated and the evaluation result corresponding to the semantic relevance of the model to be evaluated.

[0020] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0022] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this application do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0023] Semantic representation models are a class of models used to capture and represent the semantic information of text (such as words, sentences, or paragraphs). Their main goal is to convert complex linguistic data into a form that machines can understand, while preserving as much of its original semantic and contextual information as possible. These models are crucial in natural language processing tasks such as intent recognition, text classification, and information retrieval.

[0024] Embedding models are a specific way to implement semantic representation models. They learn to embed words or sentences into a dense vector space, thus grouping semantically similar objects closer together in the vector space. Embedding models are a type of embedding model that maps high-dimensional data to a low-dimensional space, commonly used in natural language processing, computer vision, and recommender systems. Embedding models can convert complex objects (such as words, sentences, or images) into vector formats that can be processed in mathematical spaces. They can also convert discrete objects (such as words) into dense vectors that capture semantic relationships between objects. Furthermore, embedding models reduce computational complexity and improve efficiency by mapping data from high-dimensional to low-dimensional spaces.

[0025] In intelligent customer service systems, embedding models play a crucial role, especially in multi-model fusion strategies. During intent recognition, the system compares the user's input embedding vector with predefined intents (also converted into vectors). By calculating the similarity between these vectors (such as cosine similarity), the system can determine which intents the user's input most closely resembles. This calculation method effectively captures the meaning of the user's input. The output of the embedding model can serve as input for other text classification technologies (such as FastText and BERT), which can further process these vectors for more complex intent predictions. By combining results from different models (such as embedding models and text classification models), intelligent customer service systems can use voting or weighted methods to determine the final intent recognition result. For example, if both the embedding model and the BERT model agree that the user's input is "querying order status," the system will make this judgment with greater confidence. This method effectively reduces the risk of misjudgment due to the insufficient performance of a single model.

[0026] Therefore, the role of embedding models in intelligent customer service systems is not limited to simple text representation; it is the foundation for achieving efficient and accurate intent recognition. By combining with other models, embedding models can enhance the overall performance of the system and improve the user experience.

[0027] However, with the rapid development of natural language processing, embedding model technology is constantly iterating, resulting in a new generation of more efficient and accurate models. These new models typically possess stronger semantic representation capabilities and demonstrate superior accuracy in similarity calculation and intent matching. Therefore, introducing these advanced embedding models into existing intelligent customer service systems usually requires addressing several technical challenges: first, how to conduct a comprehensive and scientific performance evaluation of candidate embedding models in real-world business scenarios to ensure that the new model outperforms the current one; second, how to align the distribution of the vectors generated by the embedding models to guarantee system compatibility and policy consistency after switching between old and new models; and third, how to complete the model replacement through efficient automated processes, reducing manual intervention and mitigating risks during the replacement process. These issues limit the application of embedding model updates in practical systems.

[0028] When facing these challenges, the relevant technologies firstly address the first challenge: the embedding models of most current intelligent customer service systems are not updated for extended periods after deployment. With the rapid iteration of embedding models in the field of natural language processing, the embedding models used in existing systems gradually lag behind in terms of semantic understanding and generalization capabilities. Although some existing technologies do update the embedding models, evaluating their performance is a crucial step in this process. However, existing technologies generally rely on manual verification, which is highly complex, prolongs the replacement cycle, increases labor costs, and carries a high risk of errors during the replacement process. Therefore, the relevant technologies do not effectively solve the first challenge. The first problem that this application aims to solve is how to efficiently and systematically conduct a comprehensive and scientific performance evaluation of the embedding model to ensure that the new model outperforms the current model.

[0029] Furthermore, regarding the second challenge mentioned above, the relevant technologies lack effective distribution alignment algorithms, making it difficult to guarantee the system fusion effect after model replacement. Therefore, the relevant technologies have not adequately addressed the second challenge. How to achieve alignment of the distribution of the embedding model's generated vectors to ensure system compatibility and policy consistency after switching between the old and new models is the second problem that this application's embodiments need to solve.

[0030] Furthermore, regarding the third challenge mentioned above, the related technologies rely entirely on manual testing and decision-making during the replacement of the embedding model, resulting in inefficiency, a lack of unified standards, and difficulty in quickly responding to changes in business needs. Therefore, how to complete the model replacement through an efficient automated process, reducing manual intervention and mitigating the risks during the replacement process, is the third problem that this application's embodiments aim to solve.

[0031] Regarding the first issue mentioned above, this application provides a method for evaluating and replacing a model. This method can efficiently and systematically conduct a comprehensive and scientific performance evaluation of the embedding model to ensure that the actual performance of the new model is better than the current model. Specifically, it includes: acquiring a target knowledge base applicable to a preset service scenario; acquiring a test dataset required for model evaluation based on historical interaction data and the target knowledge base; determining multiple models to be evaluated; performing performance evaluations on the multiple models to be evaluated based on the target knowledge base and the test dataset, analyzing the recall accuracy and semantic relevance of each model to obtain the performance evaluation result for each model; determining the replacement model based on the performance evaluation results of the models to be evaluated; and updating the original model to the replacement model. This method first acquires a target knowledge base applicable to a preset service scenario, providing the core data source for model evaluation. The target knowledge base provides background knowledge for the model, ensuring the relevance and accuracy of the evaluation results. The test dataset is acquired based on historical interaction data and the target knowledge base. The design of the test dataset fits the actual business scenario and can reflect the diversity of real user questions. The performance evaluation employs a multi-dimensional approach, assessing factors such as recall accuracy (whether the model accurately identifies knowledge items relevant to the user's question) and semantic relevance (whether the recalled results are semantically consistent with the user's question). This multi-dimensional analysis ensures the scientific rigor and accuracy of the model evaluation results. Finally, the original model is updated to the replacement model, a process that can be automated, efficiently and seamlessly replacing underperforming models. Therefore, this method, through a systematic and automated evaluation process and replacement mechanism, improves the efficiency of model evaluation and updates, ensuring the system maintains high efficiency and accuracy in a dynamic business environment.

[0032] Regarding the second issue mentioned above, before performing the step of updating the original model to the model to be replaced, the method in this embodiment further includes: fitting the similarity distribution between the model to be replaced and the original model based on distribution alignment; aligning the model to be replaced and the original model according to the similarity distribution between them; verifying the alignment effect of the model to be replaced after distribution alignment; and if the alignment effect meets preset conditions, then performing the step of updating the original model to the model to be replaced. Specifically, by statistically analyzing and fitting the similarity distribution between the model to be replaced and the original model, the differences in semantic vector generation between the two models can be ensured. Then, using the fitted similarity distribution, a mapping function is constructed. In this process, the similarity score generated by the model to be replaced is adjusted through this mapping function so that the distribution characteristics corresponding to the similarity score generated by the model to be replaced are as consistent as possible with the distribution characteristics corresponding to the similarity score generated by the original model. This process ensures that the vectors generated by the new model (i.e., the model to be replaced) match the original model in the feature space, enabling the new model to reduce abrupt changes in system behavior and achieve a smooth transition when replacing the original model. Therefore, the method in this embodiment ensures consistency in vector generation between the old and new models through a systematic approach, reduces the risk of model replacement, improves system compatibility and user experience, and provides a foundation for continuous model improvement.

[0033] Regarding the third issue mentioned above, the method in this embodiment updates the target knowledge base using a vector update based on the model to be replaced, resulting in an updated target knowledge base. It then obtains the result generated by the model to be replaced when a first preset number of users send access requests to the updated target knowledge base; this result includes the request error rate. If the request error rate is within a first preset range, the number of users accessing the updated target knowledge base is increased based on the first preset number, and the request error rate is continuously monitored until a preset condition is met, at which point the model to be replaced is released. The preset condition includes a request error rate within a first preset range and a number of users within a second preset range. This embodiment reduces the risks associated with launching a new model by gradually expanding the user access scope, ensuring system stability. Furthermore, the entire model replacement process follows established procedures and standards, ensuring the systematic nature and consistency of the replacement process.

[0034] The evaluation and replacement method of this model is illustrated below through specific embodiments.

[0035] See Figure 1 This provides a method for evaluating and replacing a model, which is applied to a preset service scenario. The method includes:

[0036] S11. Obtain the target knowledge base applicable to the preset service scenario.

[0037] Constructing a target knowledge base suitable for the current service scenario, which could be an intelligent customer service system, involves preprocessing a pre-defined QA knowledge base to obtain a target knowledge base that fits the current service scenario. This process may include: selecting knowledge entries that conform to configuration specifications from the pre-defined QA knowledge base to ensure the accuracy and consistency of the content; filtering out entries that are unsuitable for the current service scenario to ensure that the information in the target knowledge base can effectively meet the user's needs. Additionally, data cleaning can be performed, including removing redundant information and standardizing the format of knowledge entries to ensure consistency. Furthermore, missing knowledge entries can be supplemented based on user needs and the service scenario to improve the completeness of the knowledge base. The QA knowledge base is a structured data resource stored in the intelligent customer service system, mainly including question entries, answers, and metadata. Question entries are various questions that users may ask, which have been organized and categorized for easy system matching. Answers correspond to the standard answer or solution for each question entry, aiming to directly respond to the user's question. Metadata contains relevant information for each question entry, such as creation date, modification history, and applicable scenarios, to help maintain and update the knowledge base.

[0038] S12. Based on historical interaction data and the target knowledge base, obtain the test dataset required for model evaluation.

[0039] The test dataset required for model evaluation is obtained by extracting data from the aforementioned target knowledge base and historical interaction data. It primarily includes real user questions, recall results, and system responses. Real user questions are actual user inputs extracted from historical interaction data, ensuring they represent how users ask questions in real-world scenarios. Therefore, historical interaction data can include the following: questions actually entered by users during historical interactions with the system. These questions should cover various question types, such as general questions, ambiguous questions, and unknown questions, to ensure the test dataset comprehensively reflects questions of varying difficulty.

[0040] Next, a recall test is conducted on each real user question using the target knowledge base to obtain the recall results. Specifically, this involves: converting each real user question into a semantic vector; based on this semantic vector, searching the target knowledge base for the most similar entry to the question, and recalling related questions and their standard answers. These question entries are extracted from pre-annotated standard questions and answers in the target knowledge base. The recall results will include multiple related question entries, which can be sorted and returned to the user as the recall results.

[0041] For each real user question, a system response is generated based on the target knowledge base and historical interaction data. The system response may include the following types: Direct Answer: When the question is a standard question, the system directly finds the most matching question entry from the target knowledge base and returns the corresponding standard answer. Clarification Request: When the question is ambiguous, the system recognizes that the user's question is unclear and may return a prompt message asking the user to clarify their intent or provide more information for better understanding and answering the question. Unidentified Case: When the question is unknown and the system cannot find a relevant entry in the target knowledge base, the system will inform the user that it cannot answer the question and suggest that the user try changing their question or providing more contextual information.

[0042] Therefore, based on the difficulty of the problem and the system's response to it, the problems in the test dataset can be divided into three categories:

[0043] Common questions: These are clear and specific questions raised by users, and the system can directly find the answer and provide the standard answer.

[0044] Ambiguous issues: A user's question may have multiple interpretations that the system cannot directly understand, requiring the user to clarify the specific meaning of their question.

[0045] Unknown question: The user's question involves information or knowledge that is not included in the target knowledge base. The system cannot find the answer in the knowledge base and may suggest that the user provide more contextual information or change the way they ask the question.

[0046] The test dataset is divided into three categories: common problems, ambiguous problems, and unknown problems. This will help to analyze the performance of the model to be evaluated on different types of problems in more detail in the following text.

[0047] In this embodiment, a complete test dataset is constructed based on real user questions extracted from historical interaction data, relevant questions and answers recalled from the target knowledge base, and system-generated responses. This test dataset is comprehensive and diverse, enabling subsequent performance evaluation of multiple models to be evaluated and ensuring the scientific validity and accuracy of the evaluation results.

[0048] S13. Identify multiple models to be evaluated.

[0049] The selection of a suitable model for evaluation can be based on factors such as processing performance, service scenarios, storage, and computational costs. When determining the model to be evaluated, the desired outcome should be clearly defined, such as improving intent recognition accuracy or reducing response time. Based on this outcome, candidate models can be selected, choosing different types of embedding models according to actual business needs, such as word vector models (e.g., Word2Vec, GloVe), contextual language models (e.g., BERT, RoBERTa), and lightweight models (e.g., DistilBERT, ALBERT). Furthermore, the size of the selected candidate model needs to be determined to ensure efficient storage within the system, and the computational complexity should be considered to ensure it is suitable for the current system's computational resource constraints. The selected candidate models are then combined with the current system model to form the model to be evaluated. Different integration methods can be considered, such as weighted fusion of the outputs of multiple models, or selecting between the current model and candidate models based on specific conditions.

[0050] In this embodiment, the determined models to be evaluated include multiple models, which can be of different types and all of them can belong to the Embedding model.

[0051] S14. Based on the target knowledge base and test dataset, the performance of multiple models to be evaluated is evaluated respectively. The recall accuracy and semantic relevance of each model to be evaluated are analyzed to obtain the performance evaluation results of each model to be evaluated.

[0052] For details, see Figure 2 Step S14 specifically includes:

[0053] S141. For any model to be evaluated, semantic vectorization is performed on the knowledge items in the target knowledge base through the model to be evaluated, generating a semantic vector for each knowledge item, and storing the generated semantic vector in the vector database.

[0054] S142. Based on the vector database, generate a recall result list corresponding to the model to be evaluated. Specifically, the model to be evaluated performs semantic vectorization on real user queries in the test dataset, generating a semantic vector for each real user query. The semantic vector of each real user query is then compared with the semantic vectors of knowledge entries in the vector database to obtain knowledge entries similar to the real user queries. Based on the obtained knowledge entries similar to the real user queries, a recall result list corresponding to the model to be evaluated is generated. Here, a real user query refers to the query actually entered by the user.

[0055] S143. Compare the recalled questions in the recall results list with the standard questions in the target knowledge base to evaluate the recall accuracy of the model under evaluation. The recalled question refers to the question matched by the model under evaluation from the vector database during vector retrieval, which corresponds to a knowledge item semantically similar to the real user's question. The standard question refers to existing questions or items in the target knowledge base, which can be processed through organization, normalization, or standardization to serve as reference questions in the target knowledge base.

[0056] S144. Perform semantic relevance judgment between the recall questions in the recall result list and the standard questions in the target knowledge base to evaluate the semantic relevance of the model to be evaluated.

[0057] S145. Based on the evaluation results corresponding to the recall accuracy of the model to be evaluated and the evaluation results corresponding to the semantic relevance of the model to be evaluated, a comprehensive evaluation of the model to be evaluated is performed to obtain the performance evaluation results of the model to be evaluated.

[0058] This embodiment proposes a multi-dimensional recall evaluation method, including automated analysis of recall accuracy and semantic relevance. By constructing a standardized comparison and evaluation process, a large language model (such as LLM) is used to rank and analyze the recall results of the model to be evaluated, and combined with a relevance verification mechanism, a scientific basis is provided for subsequent model optimization and replacement.

[0059] Specifically, firstly, the content of the target knowledge base is semantically vectorized using the model to be evaluated, generating a semantic vector for each knowledge item, and then the generated semantic vectors are stored in a vector database for use in recall testing.

[0060] Then, recall tests are conducted on the models to be evaluated using the test dataset to obtain the retrieval results of each model in real-world business scenarios. This process includes: submitting questions (i.e., real user questions) from the test dataset as input to each model to be evaluated; each model processes the input real user questions to generate corresponding semantic vector representations. This step leverages the model's embedding capabilities to convert text into vector form for subsequent similarity calculations. The semantic vector generated for each question is compared for similarity with the semantic vectors corresponding to knowledge entries in the vector database. By calculating the similarity between vectors (e.g., cosine similarity), the knowledge entry most closely related to the real user question can be identified. Since the knowledge entries in the vector database are the vectorized representations of the target knowledge base, which includes question entries and answers, the semantic vector generated for each question is compared for similarity with the semantic vectors corresponding to knowledge entries in the vector database. In other words, by comparing question similarity, the question matching the real user question and its corresponding answer are found. Based on the similarity calculation results, a recall result list is generated. This list comprises knowledge items most similar to the real user's question. These knowledge items include questions or entries from the knowledge base similar to the real user's question, along with the corresponding answers or solutions. In this embodiment, the recall result lists for each model to be evaluated are recorded, and corresponding similarity scores are attached. The attached similarity score refers to the similarity value calculated between the user's question vector and each recalled item vector. This score reflects the degree of similarity between the user's question and the knowledge item.

[0061] For example, if a user asks "How do I reset my password?" and generates a question vector, then: the target knowledge base might have an entry "Password reset steps," and its vector in the vector database would be used to calculate the similarity between it and the user's question vector. Ultimately, the recall results list would record the entry "Password reset steps," along with its corresponding similarity score, such as 0.85 (indicating a high similarity).

[0062] Next, a large language model (such as LLM) is used to compare the recalled questions in the above recall result list with the standard questions in the target knowledge base to evaluate the recall accuracy of the model under evaluation. This recall accuracy measures the degree of matching between the information retrieved by the model under evaluation and the standard questions in the target knowledge base. During the comparison, it is determined whether the recalled questions accurately capture the core theme and main keywords of the standard questions, and it is also analyzed whether the recalled questions specifically reflect the details and scenarios of the standard questions. Here, the recalled questions are those similar to real user queries obtained from the target knowledge base. In this embodiment, a standardized Prompt template is designed to guide LLM in ranking the recall results of multiple models under evaluation and generating structured JSON output, such as {"ranking":"model1>model3=model2"}, to clarify the differences in recall accuracy among the various models under evaluation. "model1>model3=model2" is the ranking value, representing the difference in recall accuracy among the different models under evaluation. "model1>model3" indicates that the recall accuracy of model 1 is higher than that of model 3, meaning that model 1 performs better in capturing the core theme and main keywords of the standard questions. "model3 = model2" indicates that Model 3 and Model 2 have equal recall and accuracy, meaning that the two models perform similarly in capturing the subject and details of the standard question. Therefore, Model 1 has the most accurate recall, while Model 3 and Model 2 have the same accuracy, both lower than Model 1. This structured output format facilitates quick understanding and comparison of the performance of different models, guiding subsequent model optimization and selection.

[0063] Next, a generative large language model (such as LLM) is used to analyze and evaluate the recall results, aiming to assess their relevance. In this embodiment, a standardized Prompt template is designed to ensure consistency and repeatability in the evaluation of recall results. Based on a pre-defined verification mechanism, the LLM is guided to determine the relevance of the model's recall results. This verification mechanism includes: comparing the recalled question with the standard question; and relevance assessment. Comparing the recalled question with the standard question involves semantically comparing the recalled questions in the recall result list with the standard questions in the target knowledge base to determine whether their semantics are consistent or similar. Relevance assessment evaluates whether the recalled results in the recall result list are related to the standard questions in the target knowledge base, ensuring that the information provided to the user is useful. Through the above comparison and assessment, a structured result is generated, such as {"related":[true,false,true,true]}, where each Boolean value indicates whether the corresponding recall result is related to the standard question. The recalled question is the question or entry obtained from the target knowledge base that is similar to a real user's question, and the recalled result is the answer or solution corresponding to that question or entry. One approach is to use batch processing (e.g., verifying every 5 results) to improve the efficiency and stability of large-scale data evaluation, thereby enabling efficient processing and evaluation of multiple recall results.

[0064] In this embodiment, LLM is used to compare and evaluate the relevance of recall questions and their corresponding results, ensuring that the system can provide accurate and relevant answers or solutions in practical applications. This mechanism provides effective support for evaluating and optimizing system performance.

[0065] Next, based on the evaluation results of recall accuracy and the evaluation results of semantic relevance corresponding to the recall results, a comprehensive score is generated for each model to be evaluated. This comprehensive score is used to evaluate the performance of the model.

[0066] See Figure 3 The above step S145 specifically includes:

[0067] S1451. Based on the question categories corresponding to real user questions in the test dataset and the evaluation results corresponding to the recall accuracy of the model to be evaluated, obtain the accuracy score of the model to be evaluated for each question category.

[0068] S1452. Based on the question categories corresponding to real user questions in the test dataset and the evaluation results corresponding to the semantic relevance of the model to be evaluated, obtain the recall score of the model to be evaluated for each question category.

[0069] S1453. Obtain the weight corresponding to the problem category.

[0070] S1454. Based on the weights, accuracy score, and recall score, calculate the overall score of the model to be evaluated to obtain the performance evaluation result of the model to be evaluated.

[0071] As can be seen, the test dataset includes common questions, ambiguous questions, and unknown questions. In this embodiment, for each question in the test dataset, its performance score under different models to be evaluated is calculated. This performance score includes accuracy score, recall score, and F1 score. The accuracy score is used to evaluate the precision of the positive examples generated by the model, the recall score is used to evaluate the recall rate of the true positive examples generated by the model, and the F1 score is used to comprehensively consider the accuracy and recall scores.

[0072] After calculating the performance score of each problem in the test dataset under different evaluation models, for each evaluation model, the score of each evaluation model under the above three categories of problems is calculated, and the average value is calculated to determine the performance of the evaluation model on simple, medium, and hard problems. Among them, ordinary problems reflect the performance of the evaluation model on simple problems, ambiguous problems reflect the performance of the evaluation model on medium problems, and unknown problems reflect the performance of the evaluation model on hard problems.

[0073] For example, suppose there are multiple models to be evaluated (e.g., Model A, Model B, Model C, etc.). Each model will process three question types (normal questions, ambiguous questions, and unknown questions) from the same test dataset. Each model will answer these three types of questions and calculate a score based on metrics such as accuracy and recall. A score will be generated for each question in each question type; these scores can be accuracy, recall, or F1 score. For Model A, Model B, and Model C, their scores on the three question types are as follows: Normal questions: Model A: 0.85; Model B: 0.80; Model C: 0.90. Ambiguous questions: Model A: 0.67; Model B: 0.60; Model C: 0.75. Unknown questions: Model A: 0.45; Model B: 0.50; Model C: 0.40. Then, the average score for these three types of questions is calculated for each model. The average score for model A is (0.85 + 0.67 + 0.45) / 3 = 0.66, the average score for model B is (0.8 + 0.6 + 0.5) / 3 = 0.63, and the average score for model C is (0.9 + 0.75 + 0.4) / 3 = 0.68. The average score for each model can be used as the overall score for that model. Of course, other methods can also be used to calculate the overall score for each model. For example, the overall score of a model can be calculated by weighting the F1 scores of questions of different difficulty levels. For example, the overall score for model A is S = a1 * F1. 简单 +a2*F1中等+ a3*F1 困难 Similarly, the combined score of Model B and Model C is obtained.

[0074] For model A, its scores on the three types of questions are 0.85, 0.67, and 0.45. The average score can be calculated to obtain the total score of model A across these three types of questions. For example: Ordinary questions: Question 1: Score 0.8; Question 2: Score 0.9; Question 3: Score 0.85. Ambiguous questions: Question 1: Score 0.6; Question 2: Score 0.75; Question 3: Score 0.65. Unknown questions: Question 1: Score 0.4; Question 2: Score 0.5; Question 3: Score 0.45. The average score for the three ordinary questions is (0.8 + 0.9 + 0.85) / 3 = 0.85; the average score for the three ambiguous questions is (0.6 + 0.75 + 0.65) / 3 = 0.67; the average score for the three unknown questions is (0.4 + 0.5 + 0.45) / 3 = 0.45. Therefore, the total score of model A across the three types of questions can be obtained. Among them, Model A performed well on simple problems (i.e. ordinary problems) with a score of 0.85, performed moderately well on medium problems (ambiguous problems) with a score of 0.67, and performed poorly on difficult problems (unknown problems) with a score of 0.45.

[0075] Each model's score reflects its ability to handle different types of problems. By comparing the scores of each model, we can evaluate its overall performance, as well as its strengths and weaknesses in specific problem types, thus providing a basis for model selection and optimization.

[0076] In some embodiments, the recall results of multiple models to be evaluated can be ranked according to the accuracy evaluation results corresponding to the recall results. In this embodiment, the ranking information of the recall results of the models to be evaluated is converted into scores, and the scores are allocated to a predetermined range, such as [0,1), using a linear allocation method. For models with the same ranking, the average score is calculated to ensure fair allocation. Specifically, the initial score of the model to be evaluated is calculated using a linear difference based on the model's ranking, with the specific formula being: p i =(Nr i ) / (N-1), where p i Here is the initial score for model i to be evaluated, N is the total number of models to be evaluated, and r is the initial score. i The ranking of model i to be evaluated (the ranking can start from 1).

[0077] For models with the same ranking, calculate the average of their scores and assign it to all models with the same ranking. For example, suppose there are k models with the same ranking, each with a ranking of r and a corresponding score range of [p...].max p min The smoothing distribution formula is: p avg =(p max +...+p min ) / k.

[0078] Through the above linear interpolation calculation and smoothing processing using the smoothing assignment formula, the accuracy ranking score P = [p1, p2, ..., p] of each model to be evaluated for this problem can be output. N This ensures that the score is within the range [0,1] and satisfies the rules of smooth allocation.

[0079] The accuracy score of model i relative to question j will be obtained by combining the accuracy ranking score and the relevance result. For example, if the accuracy score of model i relative to question j is P... i,j ,

[0080] P i,j =0.5*s i,j +0.5*is_related(i,j);

[0081] Among them, s i,j This is the accuracy ranking score of model i relative to question j. This score is obtained by ranking the recall results of the models being evaluated and smoothing the distribution using linear interpolation and average distribution to ensure the model's score remains within a predetermined range. `is_related(i,j)` is a binary variable indicating whether model i is relevant to question j in the target knowledge base; for example, a value of 1 if relevant and 0 if irrelevant. Therefore, this accuracy score is a weighted average of the model's accuracy ranking score and its relevance score to the question, reflecting both the model's accuracy and relevance to the question.

[0082] The accuracy score of the model to be evaluated was calculated using the above embodiments. The recall score and F1 score of the model to be evaluated will also be calculated below.

[0083] The recall score of the model to be evaluated relative to the question can be calculated based on the relevance judgment of the recall results mentioned above. This involves calculating the proportion of relevant questions in the recall results of the model to be evaluated, i.e., the recall score R of model i relative to question j. i,j =related_count(i,j) / N rHere, related_count(i,j) represents how many questions j in the recall result list corresponding to the model i to be evaluated are related to real user questions, and the number of such related questions is related_count(i,j). r This indicates the number of questions recalled by the model to be evaluated each time. Therefore, the recall score represents the percentage of questions in the recall results list that are relevant to real user questions.

[0084] The F1 score of the model to be evaluated for the question is taken as the harmonic mean of the accuracy and recall scores, which comprehensively considers the accuracy and coverage of the model's predictions. The F1 score of model i for question j is: F1 i,j =(2*(P) i,j *R i,j )) / (P i,j +R i,j ).

[0085] In one example, the overall score S of the model to be evaluated is calculated by weighting the scores of the datasets for problems of different difficulties, that is: S i =(F1) s_i +F1 m_i +F1 h_i ) / 3, where F1 s_i F1 m_i F1 h_i These are the F1 scores of the model i to be evaluated on simple, medium, and difficult problems, respectively.

[0086] The above explains how to calculate the accuracy score, recall score, and F1 score of the model to be evaluated, where the F1 score is calculated based on the accuracy score and recall score. Next, the F1 scores of the model to be evaluated on questions of different difficulty levels are weighted and calculated to obtain the comprehensive score of the model to be evaluated. This comprehensive score is used to represent the performance evaluation result of the model to be evaluated.

[0087] For example: Suppose there are three models in the scenario (Model A, Model B, Model C), and the evaluation results for a specific problem are as follows:

[0088] Model ranking (r): Model A: 1, Model B: 2, Model C: 2 (same level);

[0089] Total number of models (N): 3;

[0090] Model recall results: Model A: 8 relevant questions, Model B: 6 relevant questions, Model C: 5 relevant questions, for a total of 10 questions recalled for each model.

[0091] Next, we perform normalized score allocation, that is, calculate the initial score P:

[0092] For Model A: p A =1-(1-1) / (3-1)=1;

[0093] For Model B: p B =1-(2-1) / (3-1)=0.5;

[0094] For Model C (at the same level as B): p C =0.5.

[0095] Next, we calculate the accuracy score, assuming the relevance judgment results of the models are: Model A: 1 (relevant), Model B: 1 (relevant), Model C: 0 (irrelevant). Therefore, we calculate the accuracy score P:

[0096] P A =p A *C A =1*1=1;

[0097] P B =p B *C B =0.5 * 1 = 0.5;

[0098] P C =p C *C C =0.5*0=0.

[0099] Next, calculate the recall score. Recall R:

[0100] R A =8 / 10=0.8; R B =6 / 10=0.6; R C =5 / 10=0.5.

[0101] Next, calculate the value of F1:

[0102] F1 A =2*A A *R A / (A A +R A )=2*1*0.8 / (1+0.8)=0.89;

[0103] F1 B =2*A B *R B / (A B +R B= 2 * 0.5 * 0.6 / (0.5 + 0.6) = 0.55;

[0104] F1 A =2*A C *R C / (A C +R C = 2*0*0.5 / (0+0.5) = 0.

[0105] Finally, a comprehensive score is calculated. Assuming the weights for easy, medium, and hard questions are 0.5, 0.3, and 0.2 respectively, the F1 scores for the three types of questions are as follows:

[0106] Simple question: Model A: 0.89, Model B: 0.55, Model C: 0

[0107] Medium-difficulty problems: Model A: 0.80, Model B: 0.50, Model C: 0

[0108] Difficult problem: Model A: 0.70, Model B: 0.45, Model C: 0

[0109] The overall score S of the model is calculated as follows:

[0110] For Model A:S A =0.5*0.89+0.3*0.8+0.2*0.7=0.825;

[0111] For Model B:S B =0.5*0.55+0.3*0.5+0.2*0.45=0.515;

[0112] For Model C:S C =0.5*0 + 0.3*0 + 0.2*0 = 0;

[0113] Therefore, the accuracy score and recall score of each model can be calculated, the F1 score can be calculated based on the accuracy and recall scores, and the final composite score can be calculated based on the F1 score. This embodiment generates a composite score for each model to be evaluated by combining the recall and accuracy ranking results with the relevance judgment results. This composite score provides a scientific basis for determining the replacement model in the following text.

[0114] S15. Based on the performance evaluation results of the model to be evaluated, determine the model to be replaced.

[0115] Based on the performance evaluation results of the model to be evaluated, the replacement model is determined. This involves considering the overall score of the model to be evaluated on problems of varying difficulty, as determined by the model's performance evaluation, and then selecting a suitable replacement model based on its usage in the business scenario. The model with the highest score can be identified as the replacement model.

[0116] S16. Update the original model to the model to be replaced.

[0117] When updating the original model to the replacement model, the goal is to ensure service stability after the replacement model replaces the original model, avoiding the risks of a direct full replacement. This primarily focuses on the number of normal service requests (whether user requests successfully return results) and the request error rate (the proportion of failed requests to total requests). The number of failed requests can be calculated based on the number of access requests sent by a preset number of users and the number of normal service requests, and then the proportion of failed requests to total requests is calculated to obtain the request error rate. Then, a canary release strategy is adopted, where the replacement model does not cover all traffic at once, but allows a subset of users to access the results of the replacement model. During this process, as long as the request error rate meets the requirements (i.e., the error rate is within an acceptable range), the replacement model is considered to have no impact on availability and can continue to be rolled out. Next, the coverage is gradually expanded until the original model is completely replaced. If the error rate is not within an acceptable range, the expansion of the user base is paused, the causes of errors are analyzed, the model is optimized, and service strategies are adjusted. In summary, this embodiment uses canary release, first testing on a small scale, monitoring the error rate, and then gradually rolling out the model if the error rate meets the requirements.

[0118] The process of updating the original model to the replacement model includes: updating the target knowledge base vectors using the replacement model to generate an updated target knowledge base; obtaining the result generated by the replacement model when a first preset number of users send access requests to the updated target knowledge base, the result including the request error rate; if the request error rate is within a first preset range, then increasing the number of users accessing the updated target knowledge base based on the first preset number, and continuously monitoring the request error rate until a preset condition is met, at which point the replacement model is released. The preset condition includes the request error rate being within the first preset range and the number of users being within a second preset range. The first preset number, the first preset range, and the second preset range can be set according to the specific application scenario.

[0119] The above process validates the model's effectiveness by continuously increasing the number of users and generating results. This process ensures a comprehensive evaluation of the model to be replaced on a larger user base, providing a solid basis for the final model switch.

[0120] Understandably, performance fluctuations are prone to occur during model replacement due to differences in the output distribution of the old and new models (e.g., differences in training data, model architecture, etc., which may cause the new model to produce different outputs for the same input). Distribution alignment ensures that the output features of the new model (i.e., the model to be replaced) are as consistent as possible with the old model (i.e., the original model), thereby reducing performance fluctuations caused by model replacement. Therefore, in some embodiments, after determining the model to be replaced, before updating the original model to the model to be replaced, it is necessary to consider how to align the model's generated vector distribution to ensure system compatibility and policy consistency after the switch between the old and new models.

[0121] Therefore, before executing step S16, which updates the original model to the model to be replaced, the method further includes: fitting the similarity distribution between the model to be replaced and the original model based on distribution alignment; performing distribution alignment between the model to be replaced and the original model according to the similarity distribution between the model to be replaced and the original model; verifying the alignment effect of the model to be replaced after distribution alignment; and if the alignment effect meets the preset conditions, then executing the step of updating the original model to the model to be replaced.

[0122] One approach is to align the similarity score distributions generated by the new and old models using statistical analysis. This process includes: First, fitting and modeling the similarity distribution. This involves analyzing the similarity score distributions generated by the new and old models to gain a deeper understanding of their performance on different inputs. By establishing a similarity distribution model, a mathematical foundation is provided for subsequent distribution alignment, ensuring that the output of the new model can be reasonably mapped to the distribution of the old model. Specifically, this includes: selecting representative questions from the test dataset and dividing them into training and test sets. The training set is used for model training, and the test set is used for validation. The new and old models are then used to perform similarity retrieval on the questions in the training set, recording the model's recall results and corresponding similarity scores to obtain the similarity score distributions of the new and old models. Statistical charts (such as histograms) can be used to visualize the similarity score distributions of the two models before fitting the similarity distribution. Kernel density estimation (KDE) can be chosen for fitting the similarity distribution. KDE is a non-parametric method that infers the entire distribution from local information between data points without assuming that the data follows a specific distribution. After this step, we can obtain a quantitative description and visualization of the similarity scores between the old and new models, which provides the necessary foundation for distribution alignment in subsequent steps.

[0123] Next, a distribution mapping function is constructed. For example, the cumulative distribution function is used first, as it reflects the cumulative probability of the distribution, helping to understand the distribution of different similarity scores. Then, an interpolation method is used to construct a distribution alignment mapping function. By mapping the cumulative distribution function of the new model to the cumulative distribution function of the old model, it is ensured that the similarity scores generated by the new model are reasonably aligned within the distribution range of the old model. Specifically, by constructing a mapping function to align the distribution of the similarity scores of the new model to the distribution of the old model, it is ensured that the similarity scores generated by the new model can reasonably reflect the distribution characteristics of the old model, thereby maintaining the stability of the system. Moreover, by aligning the distribution, performance fluctuations or inconsistencies that may be caused after model replacement can be reduced.

[0124] Therefore, by aligning the distributions, the output of the model to be replaced is ensured to be consistent with the original model, thus avoiding performance degradation caused by model replacement. Distribution alignment refers to adjusting the output distribution of one model to match the distribution of another model, thereby maintaining statistical consistency. In this embodiment, distribution alignment mainly refers to the alignment of the similarity score distributions between the original model and the model to be replaced.

[0125] To confirm whether the similarity score of the new model can be fully aligned to the distribution of the old model after mapping, in some embodiments, the above mapping function is also verified, that is: verify the alignment effect of the model to be replaced after distribution alignment. If the alignment effect meets the preset conditions, then the above step S16 is executed.

[0126] Specifically, the evaluation assesses whether the replacement model, after being mapped using the aforementioned mapping function, can effectively match the distribution of the original model. Statistical metrics (such as KL divergence) can be used to quantify the similarity between the distributions of the new and old models. The preset conditions refer to standards or thresholds set during the validation process. For example, a KL divergence (KL) less than a certain value indicates that the distribution of the new model is sufficiently close to that of the old model. If, after validation, the alignment effect of the replacement model meets the above preset conditions, it means that the new model has successfully aligned with the distribution of the original model and can replace the original model in practical applications. At this point, the operation of updating the original model to the replacement model can be performed.

[0127] The new model is applied to each question in the test set to calculate the similarity score between these questions and the standard question. This score is then substituted into the previously constructed mapping function: Saligned = T(Snew). The similarity between the aligned distribution Saligned and the old distribution Sold is compared, and the alignment result is evaluated using KL divergence. KL divergence is a metric that measures the similarity between two probability distributions; a smaller value indicates greater similarity and that the alignment is as expected. If the similarity between the two distributions is low, the alignment is not as expected and further adjustments may be needed. These adjustments could include reselecting the interpolation method, modifying the form of the mapping function, or increasing the sample size. This effectively adjusts the mapping function to improve the alignment between the new and old models. Therefore, validating the mapping function ensures the effectiveness and reliability of the new model.

[0128] It should be noted that in the above embodiments, there is no necessarily a certain order between the steps. Those skilled in the art can understand from the description of the embodiments of this application that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in turn, etc.

[0129] See Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 20 includes one or more processors 21 and a memory 22. The memory 22 is connected to one or more processors 21, for example, via a bus.

[0130] Processor 21 is configured to support the electronic device 20 in performing the corresponding functions in the methods described in the above method embodiments. Processor 21 may be a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof. The aforementioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0131] Memory 22 is used to store program code, etc. Memory 22 may include volatile memory (VM), such as random access memory (RAM); memory may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory 22 may also include combinations of the above types of memory.

[0132] The memory 22 can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the model evaluation and replacement method in the embodiments of this application. The processor 21 executes various functional applications and data processing of the model evaluation and replacement method by running the non-volatile software programs, instructions, and modules stored in the memory 22, thereby implementing the model evaluation and replacement method provided in the above method embodiments.

[0133] The one or more modules are stored in the memory 22, and when executed by the one or more processors 21, they perform the model evaluation and replacement method in any of the above method embodiments.

[0134] The electronic devices in this application embodiment may specifically be ultra-mobile personal computer devices, smart displays or all-in-one machines, servers or server clusters, etc.

[0135] This application provides a non-volatile computer-readable storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 4 One of the processors 21 can enable the one or more processors to execute the model evaluation and replacement method in any of the above method embodiments.

[0136] This application provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions that, when executed by the electronic device, enable the electronic device to perform the model evaluation and replacement method in any of the above method embodiments.

[0137] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0138] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A method for evaluating and replacing a model, wherein the model is applied to a preset service scenario, characterized in that, The methods include: Obtain the target knowledge base applicable to the preset service scenario; Based on historical interaction data and the target knowledge base, obtain the test dataset required for model evaluation; The historical interaction data includes real user questions, which are questions that users actually input during the historical interaction process with the system. Identify multiple models to be evaluated; Based on the target knowledge base and the test dataset, the performance of the multiple models to be evaluated is evaluated respectively. The recall accuracy and semantic relevance of each model to be evaluated are analyzed to obtain the performance evaluation result of each model to be evaluated. The recall accuracy is used to measure the matching degree between the information retrieved by the model to be evaluated and the standard questions in the target knowledge base. The semantic relevance is used to measure the matching degree between the results generated by the model to be evaluated and the semantics of the user's question. Based on the performance evaluation results of the model to be evaluated, determine the model to be replaced; Update the original model to the model to be replaced.

2. The method according to claim 1, characterized in that, The test dataset required for model evaluation, obtained based on historical interaction data and the target knowledge base, includes: Obtain real user questions based on historical interaction data; The real user question is converted into a semantic vector, and entries related to the real user question are searched in the target knowledge base based on the semantic vector. The entries are used as the recall results. The system response results corresponding to the real user's question are generated based on the target knowledge base and the historical interaction data. The test dataset consists of real user questions, recall results, and system response results.

3. The method according to claim 2, characterized in that, The real user questions include general questions, ambiguous questions, and unknown questions.

4. The method according to claim 1, characterized in that, The performance evaluation of the multiple models to be evaluated is performed based on the target knowledge base and the test dataset, and the recall accuracy and semantic relevance of each model to be evaluated are analyzed to obtain the performance evaluation result of each model to be evaluated, including: For any model to be evaluated, the knowledge items in the target knowledge base are semantically vectorized using the model to be evaluated, generating a semantic vector for each knowledge item, and the generated semantic vector is stored in a vector database. Based on the vector database, a recall result list corresponding to the model to be evaluated is generated; The recalled questions in the recall result list are compared with the standard questions in the target knowledge base to evaluate the recall accuracy of the model to be evaluated. The semantic relevance of the recalled questions in the recall result list to the standard questions in the target knowledge base is judged to evaluate the semantic relevance of the model to be evaluated. Based on the evaluation results corresponding to the recall accuracy and the semantic relevance of the model to be evaluated, the model to be evaluated is comprehensively evaluated to obtain the performance evaluation result of the model to be evaluated.

5. The method according to claim 4, characterized in that, The step of generating a recall result list corresponding to the model to be evaluated based on the vector database includes: The model to be evaluated performs semantic vectorization on the real user questions in the test dataset to generate a semantic vector for each real user question. The semantic vector of each real user's question is compared with the semantic vector of the knowledge item in the vector database to obtain the knowledge item that is similar to the real user's question. Based on the knowledge items obtained that are similar to the real user's question, a recall result list corresponding to the model to be evaluated is generated.

6. The method according to claim 4, characterized in that, The step of determining the semantic relevance of the recalled questions in the recall result list with the standard questions in the target knowledge base to evaluate the semantic relevance of the model to be evaluated includes: The recalled questions in the recall result list are semantically compared with the standard questions in the target knowledge base to determine whether the semantics of the recalled questions and the standard questions are consistent or similar; and, the recall results in the recall result list are evaluated to determine whether they are related to the standard questions in the target knowledge base.

7. The method according to claim 4, characterized in that, The step of comprehensively evaluating the model to be evaluated based on the evaluation results corresponding to the recall accuracy and the semantic relevance of the model to be evaluated, to obtain the performance evaluation result of the model to be evaluated, includes: Based on the question categories corresponding to real user questions in the test dataset and the evaluation results corresponding to the recall accuracy of the model to be evaluated, an accuracy score is obtained for the model to be evaluated for each question category. Based on the question categories corresponding to real user questions in the test dataset and the evaluation results corresponding to the semantic relevance of the model to be evaluated, the recall score of the model to be evaluated for each question category is obtained. Obtain the weight corresponding to the problem category; Based on the weights, the accuracy score, and the recall score, a comprehensive score is calculated for the model to be evaluated to obtain the performance evaluation result of the model to be evaluated.

8. The method according to claim 1, characterized in that, Before performing the step of updating the original model to the model to be replaced, the method further includes: Based on distribution alignment, fit the similarity distribution between the model to be replaced and the original model; Based on the similarity distribution between the model to be replaced and the original model, the model to be replaced and the original model are aligned in terms of distribution. Verify the alignment effect of the model to be replaced after distribution alignment. If the alignment effect meets the preset conditions, then execute the step of updating the original model to the model to be replaced.

9. The method according to claim 1, characterized in that, The step of updating the original model to the model to be replaced includes: The target knowledge base is updated by vector update using the model to be replaced, thereby generating the updated target knowledge base; When a first preset number of users send access requests to the updated target knowledge base, the result generated by the model to be replaced includes the request error rate. If the request error rate is within a first preset range, the number of users accessing the updated target knowledge base will continue to increase based on the first preset number, and the request error rate will be continuously monitored until a preset condition is met, at which point the release of the model to be replaced will be completed. The preset condition includes the request error rate being within the first preset range and the number of users being within the second preset range.

10. An electronic device, characterized in that, include: A memory and a processor, the memory being connected to the processor, the processor being configured to execute one or more computer programs stored in the memory, the processor causing the electronic device to perform the method as described in any one of claims 1-9 when executing the one or more computer programs.

Images