Intelligent drill evaluation method and system based on large model and RAG, medium, program product and terminal

Abstract

The invention provides an intelligent drill evaluation method and system based on a large model and RAG, a medium, a program product and a terminal. The method comprises the steps that a virtual training scene is constructed based on a large language model; the trainee carries out dialogue interaction with the virtual customer portrait in the virtual training scene, and dialogue interaction data is collected in real time; and analyzing and feeding back the virtual customer and the dialogue interaction data by adopting RAG, and evaluating the comprehensive ability of the trainee based on a multi-dimensional evaluation index. According to the invention, trainees are helped to identify portrait characteristics of virtual customers, determine questions possibly concerned by the portrait customers and provide answering strategies; and the professional ability of the trainee is comprehensively evaluated from multiple dimensions of answer fluency, objection question processing, portrait question pertinence, customer emotion processing, insurance professional knowledge and the like, so that the accuracy and fairness of evaluation are ensured. According to the method, virtual customer portrait analysis is combined with insurance industry information and enterprise private domain knowledge, the professional ability and service quality of trainees can be comprehensively improved, and a multi-dimensional evaluation mechanism is of great significance to improvement of customer service quality and satisfaction.

Description

Technical Field

[0001] The present application relates to the field of intelligent drill evaluation technology, and in particular to an intelligent drill evaluation method, system, medium, program product and terminal based on a large model and RAG. Background Art

[0002] At present, in the insurance agent training system, clearance drills are a key link in improving sales capabilities. With the development of technology, the drill model is changing from traditional offline drills to the first-generation online drill model. This change has brought some obvious advantages, but there are also some problems, especially the first-generation online drill model has significant limitations in the evaluation process, which seriously restricts the improvement of training effects and the development of agent capabilities.

[0003] First, the first generation of online practice evaluation standards are relatively rigid, focusing only on the correctness of the text, lacking flexibility and personalization. This evaluation method cannot be tailored to the individual and the situation, and cannot adapt to the specific needs and styles of different agents. For example, traditional evaluation may not be able to distinguish the differences in agent performance in different sales scenarios, or provide customized feedback based on the agent's personal development stage.

[0004] Secondly, the evaluation model in the first-generation online drill model lacks real-time and dynamic features. In actual sales drills, agents may need immediate feedback to adjust strategies and techniques, but the evaluation in the first-generation online drills often provides feedback only after the drill is over, which limits the agent's ability to learn and improve immediately.

[0005] Furthermore, the first-generation online training model cannot provide personalized and differentiated feedback. Each agent has his or her own learning characteristics and needs, but the first-generation online evaluation system may adopt a "one-size-fits-all" approach and cannot give differentiated feedback based on the trainees' performance, affecting the personalization and effectiveness of training.

[0006] In summary, the existing insurance agent training system has problems such as lack of personalization, real-time and dynamic evaluation methods during the exercise process. These problems not only affect the training effect of agents, but also limit the company's accurate evaluation and optimization of training effects. Summary of the invention

[0007] In view of the above-mentioned shortcomings of the prior art, the present invention provides an intelligent drill evaluation method, system, medium, program product and terminal based on a large model and RAG, which are used to solve the problems of lack of personalization, real-time and dynamics in the evaluation method of the insurance agent training system in the prior art during the drill.

[0008] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides an intelligent rehearsal evaluation method based on a big model and RAG, including: constructing a virtual training scenario based on a big language model; the trainer interacts with the virtual customer portrait in the virtual training scenario and collects the conversation interaction data in real time; using RAG to analyze and feedback the virtual customer portrait and the conversation interaction data, and evaluating the comprehensive ability of the trainer based on multi-dimensional evaluation indicators.

[0009] In some embodiments of the first aspect of the present application, the multi-dimensional evaluation indicators include: answer fluency, objection question handling, portrait question targeting, customer emotion processing and insurance expertise.

[0010] In some embodiments of the first aspect of the present application, the process of using RAG to analyze and feedback the conversation interaction data includes: vectorizing the conversation interaction data to obtain vectorized conversation interaction data; using RAG to query and retrieve the vectorized conversation interaction data in a pre-built insurance industry information and enterprise private domain information knowledge base to obtain knowledge text that meets preset relevant requirements; forming a prompt template based on the knowledge text and conversation interaction data that meet the preset relevant requirements; inputting the prompt template into the virtual training scene to generate a conversation result corresponding to the conversation interaction data; and feeding back the conversation result corresponding to the conversation interaction data to the trainer.

[0011] In some embodiments of the first aspect of the present application, the method further includes: after feeding back the conversation result corresponding to the conversation interaction data to the trainer, dynamically adjusting the conversation strategy of the trainer to optimize the conversation interaction data according to the conversation strategy.

[0012] In some embodiments of the first aspect of the present application, the construction process of the pre-constructed insurance industry information and enterprise private domain information knowledge base includes: obtaining insurance industry information and enterprise private domain knowledge, processing the insurance industry information and enterprise private domain knowledge to generate an initial knowledge base; using a pre-trained embedding model to vectorize the initial knowledge base to obtain a vectorized initialization knowledge base; storing the vectorized initialization knowledge base in a vector database to form an insurance industry information and enterprise private domain information knowledge base.

[0013] In some embodiments of the first aspect of the present application, the method further includes: evaluating the comprehensive ability of the trainee based on multi-dimensional evaluation indicators to obtain an evaluation result, and visually displaying the evaluation result.

[0014] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides an intelligent rehearsal evaluation system based on a big model and RAG, including: a virtual training scenario construction module, used to construct a virtual training scenario based on a big language model; a dialogue interaction data collection module, used for trainers to conduct dialogue interactions with virtual customer portraits in the virtual training scenario, and collect dialogue interaction data in real time; a comprehensive evaluation module, used to use RAG to analyze and feedback the virtual customer portraits and the dialogue interaction data, and evaluate the comprehensive capabilities of the trainers based on multi-dimensional evaluation indicators.

[0015] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the intelligent rehearsal evaluation method based on a large model and RAG is implemented.

[0016] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer implements the intelligent rehearsal evaluation method based on the large model and RAG.

[0017] To achieve the above-mentioned purpose and other related purposes, the fifth aspect of the present application provides an electronic terminal, including a memory, a processor and a computer program stored in the memory; the processor executes the computer program to implement the intelligent rehearsal evaluation method based on the large model and RAG.

[0018] As described above, the intelligent drill evaluation method, system, medium, program product and terminal based on large model and RAG provided by the present application have the following beneficial effects:

[0019] The present invention not only helps trainers identify issues that virtual customers may be concerned about and provide answering strategies, but also comprehensively evaluates the professional capabilities of trainers from multiple dimensions such as answer fluency, handling of objection issues, targeted questioning, customer emotion processing, and insurance expertise to ensure the accuracy and fairness of the evaluation. The present invention helps to comprehensively improve the professional capabilities and service quality of trainers by combining virtual customer portrait analysis with insurance industry information and corporate private domain knowledge, and the multi-dimensional evaluation mechanism is of great significance for improving customer service quality and satisfaction. The virtual training scenario based on large models and RAG provided by the present invention can improve the effect of drills, improve clearance efficiency, and retain training data so that insurance agents can receive effective training in simulated conversations, thereby improving their insurance sales capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1Shown is a flow chart of an intelligent drill evaluation method based on a large model and RAG in one embodiment of the present application.

[0021] Figure 2 Shown is a flow chart of using RAG to analyze and provide feedback on the conversation interaction data in one embodiment of the present application.

[0022] Figure 3 Shown is a flowchart of the construction process of a pre-built insurance industry information and enterprise private domain information knowledge base in one embodiment of the present application.

[0023] Figure 4 Shown is a structural diagram of an intelligent drill evaluation system based on a large model and RAG in one embodiment of the present application.

[0024] Figure 5 Shown is a schematic diagram of the structure of an electronic terminal in one embodiment of the present application. DETAILED DESCRIPTION

[0025] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0026] Before further describing the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are applicable to the following interpretations:

[0027] <1> Large Language Model (LLM): Large Language Model is a natural language processing technology based on deep learning, which aims to train large models that can process and generate natural language text. The core idea of ​​LLM technology is to use deep neural networks to pre-train models through large-scale text data, and use these pre-trained models to fine-tune or directly apply downstream tasks. LLM can learn rich language knowledge and language patterns from large-scale text data, so that the model can understand and generate the semantics, grammar, etc. of natural language, and has stronger language processing capabilities. LLM technology has broad application prospects in the field of natural language processing.

[0028] <2> Retrieval Augmented Generation (RAG): Retrieval Augmented Generation is a strategy to provide external knowledge sources for large models. It enables large models to retrieve from a specified knowledge base and combine contextual information to generate relatively high-quality responses, thereby reducing the model hallucination problem. This technology combines retrieval and generation to improve the performance of large language models in complex tasks.

[0029] To facilitate understanding of the embodiments of the present application, first Figure 1 Detailed description. Figure 1 A flowchart of an intelligent drill evaluation method based on a large model and RAG in an embodiment of the present invention is shown, and the method includes:

[0030] Step S11: construct a virtual training scenario based on a large language model; the large language model may include but is not limited to the following models: InternLM model, ChatGLM model, ChatGPT model, LLaMA model, etc.

[0031] It needs to be explained that in the insurance industry, improving the sales ability of insurance agents is crucial to improving performance. Traditional agent training methods often focus on theoretical learning and role-playing, but these methods have limitations in simulating real customer interactions. In order to overcome these limitations, this embodiment proposes a method for constructing a virtual training scenario using a large language model. This method collects basic information about customers, generates random customer portraits, and constructs a variety of virtual training scenarios based on this. By simulating dialogue training in virtual training scenarios, insurance agents can practice improvisation in response to various customer situations in a safe environment, thereby enhancing their communication skills and problem-solving abilities.

[0032] This approach not only improves the insurance agents’ ability to respond, but also helps them to more accurately understand and meet the needs of customers when facing different types of customers. In general, this virtual training scenario based on a large language model provides insurance agents with an efficient and practical training tool to improve their sales and service quality.

[0033] Step S12: The training personnel interact with the virtual customer portrait in the virtual training scenario, and collect the conversation interaction data in real time.

[0034] It should be noted that in the virtual training scenario, the trainer (i.e., insurance agent) interacts with the virtual customer portrait in real time to simulate a real customer service scenario. At this time, the conversation interaction data of the entire conversation process will be collected in real time, including the content of the conversation, the response of the trainer, the response of the virtual customer portrait, etc., for subsequent analysis and evaluation. Virtual customer portraits can be designed to have different personalities, needs, and attitudes, and according to the settings, use large language models to give vivid and realistic conversation responses, so that trainers can practice in a variety of situations. Virtual training scenarios provide trainers with a risk-free environment, allowing them to freely try and learn without worrying about adverse effects on real customers. Through continuous simulation training, trainers can improve their comprehensive ability levels such as communication skills, product knowledge, sales strategies, and customer service.

[0035] Step S13: Use RAG to analyze and provide feedback on the virtual customer portrait and the conversation interaction data, and evaluate the comprehensive ability of the trainer based on multi-dimensional evaluation indicators

[0036] It should be noted that in this embodiment, the virtual customer portrait is deeply mined and analyzed. By combining RAG technology with the capabilities of a large language model, the virtual customer portrait in the virtual training scenario can be deeply analyzed, including the customer's basic information, needs, preferences, etc. Through the deep mining of this information, the model can better understand the customer's behavior patterns and needs, thereby providing more accurate data support for subsequent evaluation and feedback.

[0037] Furthermore, by using the big model and RAG's analytical capabilities, we can deeply explore the virtual customer portrait, and at the same time, combined with insurance industry data, we can identify the potential problems and needs of virtual customers. This data-based in-depth analysis provides trainers with real market insights, allowing them to better understand customer needs during simulated conversations. By analyzing insurance industry data, we can identify the purchasing habits and preferences of specific customer groups, so that corresponding scenarios can be set up in virtual training scenarios, allowing trainers to practice how to respond to these specific customer needs.

[0038] In one embodiment, if Figure 2 As shown, the process of using RAG to analyze and feedback the dialogue interaction data includes:

[0039] Step S131: vectorize the conversation interaction data to obtain vectorized conversation interaction data. The recorded conversation interaction data is first preprocessed, including removing irrelevant information, correcting typos, standardizing language expressions, etc., and then vectorizing the preprocessed conversation interaction data to capture deep semantic information in the conversation. Preprocessing and vectorization can improve the accuracy and efficiency of subsequent analysis of conversation interaction data, thereby helping insurance agents better understand and handle customer needs and improve their business capabilities.

[0040] Step S132: Use RAG to query and retrieve the vectorized conversation interaction data in the pre-built insurance industry information and enterprise private domain information knowledge base to obtain knowledge text that meets the preset relevant requirements. Query retrieval methods include similarity retrieval and full-text retrieval. Similarity retrieval finds the most matching content by calculating the similarity between the conversation interaction data vector and the vector in the pre-built insurance industry information and enterprise private domain information knowledge base. Commonly used similarity calculation methods include cosine similarity, which measures the degree of similarity between two vectors in direction. Full-text retrieval is to find keywords or phrases in text data to quickly locate documents containing specific keywords.

[0041] In this embodiment, RAG technology is used for query retrieval, and knowledge texts or historical conversation records that are semantically similar to the conversation interaction data vector can be searched in the pre-constructed insurance industry information and enterprise private domain information knowledge base. The retrieved similar knowledge texts or historical conversation records can be sorted and selected according to the similarity, and the required knowledge texts or historical conversation records can be selected as the retrieval results according to the preset relevant requirements, that is, the knowledge texts that meet the preset relevant requirements. Among them, the preset relevant requirements can be the most relevant knowledge text, or all relevant knowledge texts within the set similarity range. The preset relevant requirements can be set according to the actual situation, and are not limited in this embodiment. Using RAG technology to filter out context related to conversation interaction data from the pre-constructed insurance industry information and enterprise private domain information knowledge base can eliminate interference from irrelevant information and improve the efficiency and accuracy of subsequent LLM execution.

[0042] In some examples, such as Figure 3 As shown, the construction process of the pre-built insurance industry information and enterprise private domain information knowledge base includes:

[0043] Step S1321: Obtain insurance industry information and enterprise private domain knowledge, and process the insurance industry information and enterprise private domain knowledge to generate an initial knowledge base.

[0044] It should be noted that the insurance industry information and enterprise private domain information knowledge base includes insurance industry information and enterprise private domain knowledge. Insurance industry information refers to the insurance industry's professional knowledge, sales skills, historical successful conversation records, and strategies for answering common questions. This information provides insurance agents with necessary background knowledge to help them better understand industry dynamics and customer needs.

[0045] Enterprise private domain knowledge refers to the knowledge and information accumulated by the enterprise for its specific customer groups, which includes the real data of customers, business processes and detailed records of interactions with customers. When building a knowledge base, customer information in the enterprise private domain knowledge needs to be desensitized to protect customer privacy and meet compliance requirements. Simulated customer data is constructed based on real customer data. These simulated data are similar to real data in statistical characteristics, but do not contain any information that can identify personal identities. In this way, enterprises can use it for internal training and business analysis without disclosing customer privacy. Using desensitized simulated customer data, enterprises can create more realistic and specific training scenarios to help employees better understand and master business processes, customer service skills and sales strategies. This training method can not only improve the professional ability and service level of insurance personnel, but also improve the effectiveness of internal training in enterprises. At the same time, this customized knowledge base can be optimized according to the specific business processes and customer data of the enterprise, making the training more targeted.

[0046] The insurance industry information is combined with the enterprise's private domain knowledge, so that the constructed insurance industry information and enterprise private domain information knowledge base can both protect customer privacy and be practical. Especially when training new employees, it can provide more specific and relevant cases and strategies to help them adapt to the work environment and customer needs more quickly.

[0047] Therefore, this embodiment is not only industry-wide, but can also provide customized analysis and suggestions for any company's specific products, services, and sales strategies. This customized analysis can help students better understand the company's unique value proposition and effectively convey this information in simulated conversations.

[0048] Specifically, the collected insurance industry information and enterprise private domain knowledge are subjected to data cleaning, text segmentation and structured organization to generate an initial knowledge base. Data cleaning includes removing noise, duplicates and irrelevant information to ensure the quality and accuracy of the data. Noise removal includes clearing format errors, spelling errors, etc. in documents. Removing duplicates is to ensure that the information in the knowledge base is unique and avoid duplicate storage. Removing irrelevant information is to delete content that is not directly related to insurance sales, such as internal management processes. Text segmentation is to divide long documents into multiple text blocks, taking logical coherence into consideration when segmenting, ensuring that each text block is a meaningful whole, so that information can be processed and retrieved more efficiently later. Structured organization is to organize the collected information by topic or category, such as insurance products, customer service, market analysis, etc., and knowledge graphs, FAQ formats, classification directories, etc. can be used to structure information.

[0049] Step S1322: vectorize the initial knowledge base using a pre-trained embedding model to obtain a vectorized initialization knowledge base. The embedding model uses Word2Vec, BERT or GPT series, etc.

[0050] The initial knowledge base is vectorized using a pre-trained embedding model to convert the text into vector form. These vectors are able to capture the contextual relationships and core meanings of sentences, so that semantically similar sentences can be identified by calculating the differences between vectors. The pre-training process of the embedding model is to first collect a large amount of text data, and then use this data to train the embedding model. The data should be as diverse as possible to cover different contexts and language usage. For example, the pre-training of the Word2Vec embedding model can use the CBOW model or the Skip-gram model. The CBOW model predicts the central word through the context words, and the Skip-gram model predicts the context words through the central word. The vector representation of the vocabulary can be learned using the CBOW model or the Skip-gram model.

[0051] Step S1323: The vectorized initialization knowledge base is stored in a vector database to form an insurance industry information and enterprise private domain information knowledge base. The vector database includes Faiss library, Milvus library, Chroma library and Elasticsearch library, etc., which can be selected based on business scenarios, hardware, performance requirements and other factors, and are not limited in this embodiment.

[0052] After all knowledge documents in the initial knowledge base are vectorized, an index is constructed and stored in a suitable vector database. A vector database is a system specially designed for storing and retrieving vector data, which optimizes the efficiency of processing and storing large-scale vector data. Through vectorization processing and indexing, the accuracy and efficiency of data analysis can be significantly improved, helping insurance agents better understand and handle customer needs.

[0053] Step S133: A prompt template is formed based on the knowledge text and the dialogue interaction data that meet the preset related requirements. The knowledge text that meets the preset related requirements obtained by retrieval is combined with the original dialogue interaction data to form a prompt template. The prompt template combines the retrieved information and the original dialogue to provide rich context, so that the output of the subsequent large model can more accurately reflect the current customer needs.

[0054] Step S134: inputting the prompt template into the virtual training scene to generate a dialogue result corresponding to the dialogue interaction data.

[0055] Step S135: Feedback the dialogue result corresponding to the dialogue interaction data to the training personnel.

[0056] It should be noted that the pre-built insurance industry information and enterprise private domain information knowledge base in this embodiment can be included in the virtual training scene. When the virtual customer in the virtual training scene asks a question, not only can the RAG technology analyze and understand the literal meaning of the question, but also the related insurance industry information and enterprise private domain information knowledge base content can be retrieved in real time through the RAG technology to accurately capture the core of the problem, that is, to obtain the knowledge text that meets the preset relevant requirements. Then, the prompt template formed by combining the knowledge text that meets the preset relevant requirements and the dialogue interaction data is an enhanced prompt template. The template is input into the virtual training scene to prompt the generation of more accurate results, that is, to generate the dialogue results corresponding to the dialogue interaction data. The dialogue results generated by the virtual training scene through the RAG technology are fed back to the trainers, which can be used for reference or direct adoption by the trainers during the dialogue interaction process to answer virtual customers.

[0057] In some examples, the method further includes: after feeding back the dialogue result corresponding to the dialogue interaction data to the trainer, dynamically adjusting the dialogue strategy of the trainer to optimize the dialogue interaction data according to the dialogue strategy.

[0058] It should be understood that in the virtual training scenario, while the trainer is interacting with different types of virtual customer portraits, the conversation interaction data is collected in real time. At the same time, RAG technology is used in real time to conduct in-depth mining and analysis of virtual customer portraits and analyze conversation interaction data to obtain the needs of virtual customers, and then generate reference answers (i.e., the conversation results corresponding to the conversation interaction data) and feedback them to the trainer for reference or use. The trainer adjusts his or her conversation strategy in real time based on the feedback reference answer, and optimizes his or her plan to answer the virtual customer's questions based on the conversation strategy. The conversation interaction data of the entire conversation interaction process is also continuously optimized to improve the learning efficiency of the trainer.

[0059] It is important to emphasize that real-time feedback is provided to trainers throughout the training process, and the feedback can be adjusted dynamically based on the trainer's performance. Trainers can immediately understand their performance based on the real-time feedback mechanism, and make real-time adjustments based on the feedback to optimize their answers to virtual customers. This flexibility ensures that trainers can quickly adapt and grow in changing situations, while instant loop learning greatly improves the learning efficiency of trainers.

[0060] Furthermore, in this embodiment, the dialogue results corresponding to the dialogue interaction data of the virtual training scene can be evaluated to determine whether the expected requirements are met, and the evaluation can also be based on multi-dimensional evaluation indicators. If the expected requirements are not met, the virtual training scene needs to be improved, and the parameters and algorithms in the insurance industry information and enterprise private domain information knowledge base or RAG can be adjusted according to the feedback, including retraining the embedding model, optimizing the vector database structure, adjusting the retrieval strategy, etc.

[0061] The virtual training scenario constructed in this embodiment provides a virtual customer portrait for trainers to talk with, so that trainers can improve their ability to deal with various complex situations in a safe and stress-free environment, which can help trainers obtain the required information more quickly, thereby more effectively handling customer inquiries, improving customer satisfaction and service quality. At the same time, it also undertakes monitoring, analysis and guidance functions to ensure that trainers can grow in each dialogue interaction.

[0062] The virtual training scenario based on large models and RAG provided by the present invention can improve the rehearsal effect, improve the clearance efficiency, and retain the training data so that insurance agents can receive effective training in simulated dialogues, thereby improving their insurance sales capabilities.

[0063] In one embodiment, the multi-dimensional evaluation indicators include: answer fluency, handling of objection questions, targeting of portrait questions, handling of customer emotions and insurance expertise.

[0064] It should be explained that this embodiment not only helps trainers identify issues that virtual customers may be concerned about and provide answering strategies, but also can significantly improve the quality of conversations through the application of RAG technology, avoid stereotyped repetition, and achieve the improvement of insurance agents' professional soft skills and hard power, especially from the fluency of answers, handling of objections, targeted portrait questions, customer emotion processing and insurance professional knowledge. Comprehensive assessment is conducted from multiple dimensions, and the comprehensive ability of trainers is scored and evaluated.

[0065] If the answer fluency score is too low, relevant language models and knowledge bases can be retrieved to generate more natural and fluent conversation content, and then help trainers learn how to construct coherent and logical answers, thereby improving the naturalness and professionalism of the conversation.

[0066] If the ability to handle objection issues is insufficient, the virtual training scenario provides strategic suggestions based on historical data and successful cases when facing customer objections. The objection issue handling evaluation module in the virtual training scenario can help trainers learn how to effectively identify customer concerns and provide targeted answers to resolve customer objections.

[0067] If the targeted score of the portrait questions is too low, the training staff will be guided to ask more precise questions based on the characteristics of the virtual customer portrait, which will help to further explore the needs of the virtual customers and guide the conversation in a direction that is more conducive to solving the problem.

[0068] If the customer's emotion processing score is low, the analysis of the customer's language and behavior patterns will help trainers identify the customer's emotional state and provide corresponding emotion management strategies. Customer emotion processing ability is essential to building good customer relationships and improving customer satisfaction.

[0069] If insurance expertise is insufficient, the virtual training scenario combines the company's private domain knowledge base to provide professional insurance knowledge and information to trainers. This not only enhances the professional knowledge reserves of the trainees, but also improves their ability to provide professional advice during the conversation.

[0070] It should be noted that the multi-dimensional evaluation not only focuses on the performance of trainees, but also combines the best practices and expertise of the insurance industry. Through the five-dimensional evaluation system, the trainees' abilities can be comprehensively evaluated and professional feedback and suggestions can be provided. This evaluation system includes answer fluency, handling of objection questions, targeted questioning, handling of customer emotions and insurance expertise. Each dimension has clear scoring criteria and detailed explanations to ensure the accuracy and fairness of the evaluation.

[0071] The multi-dimensional evaluation indicators of this embodiment are different from traditional evaluation criteria. They focus more on the overall performance and actual coping ability of trainees, and encourage trainees to develop more comprehensive and flexible sales skills. In the evaluation process, we not only focus on whether the trainees use the correct sales words, but more importantly, we evaluate how they flexibly adjust strategies according to different virtual customer situations and how to build trust and relationships in the dialogue.

[0072] In one embodiment, the method further includes: evaluating the comprehensive ability of the trainer based on multi-dimensional evaluation indicators to obtain an evaluation result, and visually displaying the evaluation result.

[0073] It should be noted that the comprehensive ability assessment results of trainers in the virtual training scenario are compiled into a comprehensive assessment report. This report should include information such as a summary of the trainers' performance in various aspects, specific scores, and suggestions for improvement. The report can be presented in a visual form, such as a radar chart, so as to more intuitively display the performance of the trainers. Drawing the five scores of the multi-dimensional evaluation indicators into a radar chart not only intuitively shows the comprehensive ability of the trainees, but also highlights their professional level in various fields. This professional visualization tool helps trainees and coaches quickly identify strengths and room for improvement. For example, the design of the radar chart takes into account readability and information content, making complex data clear at a glance, which is convenient for trainers to reflect on themselves and coaches to guide.

[0074] It should be emphasized that the intelligent drill evaluation method based on large models and RAG proposed in this embodiment not only helps trainers identify issues that virtual customers may be concerned about and provide answering strategies, but also comprehensively evaluates the professional capabilities of trainers from multiple dimensions such as answer fluency, objection problem handling, portrait question targeting, customer emotion processing, and insurance expertise to ensure the accuracy and fairness of the evaluation. The present invention combines virtual customer portrait analysis with insurance industry information and corporate private domain knowledge, which helps to comprehensively improve the professional capabilities and service quality of trainers, and the multi-dimensional evaluation mechanism is of great significance to improving customer service quality and satisfaction.

[0075] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects, and do not limit their order. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0076] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" represent examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0077] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, ab, ac, bc or abc, where a, b, c can be single or multiple.

[0078] Figure 4 4 is a schematic block diagram of an intelligent drill evaluation system 400 based on a large model and RAG provided in an embodiment of the present application. The system includes:

[0079] A virtual training scenario construction module 401 is used to construct a virtual training scenario based on a large language model;

[0080] A dialogue interaction data collection module 402 is used for the training personnel to conduct dialogue interactions with the virtual customer portrait in the virtual training scenario and collect dialogue interaction data in real time;

[0081] The comprehensive evaluation module 403 is used to analyze and provide feedback on the virtual customer portrait and the conversation interaction data using RAG, and to evaluate the comprehensive capabilities of the trainer based on multi-dimensional evaluation indicators.

[0082] It should be understood that the specific process of each module executing the above corresponding steps has been described in detail in the above embodiments, and for the sake of brevity, it will not be repeated here.

[0083] It should also be understood that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0084] Figure 5 is a schematic block diagram of an electronic terminal provided in an embodiment of the present application. Figure 5 As shown, the electronic terminal includes: at least one processor 501, a memory 502, at least one network interface 503 and a user interface 505. The various components in the device are coupled together through a bus system 504. It can be understood that the bus system 504 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 504 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 5 In the specification, various buses are labeled as bus systems.

[0085] The user interface 505 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0086] It is understood that the memory 502 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0087] The memory 502 in the embodiment of the present invention is used to store various categories of data to support the operation of the electronic terminal 500. Examples of these data include: any executable program for operating on the electronic terminal 500, such as an operating system 5021 and an application 5022; the operating system 5021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 5022 may include various applications, such as a media player (Media Player), a browser (Browser), etc., for implementing various application services. The intelligent drill evaluation method based on a large model and RAG provided in the embodiment of the present invention may be included in the application 5022.

[0088] The method disclosed in the above embodiment of the present invention can be applied to the processor 501, or implemented by the processor 501. The processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 501 or the instruction in the form of software. The above processor 501 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 501 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 501 may be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0089] In an exemplary embodiment, the electronic terminal 500 may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.

[0090] According to the method provided in the embodiments of the present application, the present application also provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes the intelligent rehearsal evaluation method based on a large model and RAG in any of the embodiments shown.

[0091] According to the method provided in the embodiments of the present application, the present application also provides a computer-readable storage medium, which stores program code. When the program code runs on a computer, the computer executes the intelligent rehearsal evaluation method based on a large model and RAG in any of the embodiments shown.

[0092] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program and / or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process and / or an execution thread, and a component may be located on a computer and / or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may, for example, communicate through local and / or remote processes according to signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).

[0093] Those of ordinary skill in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0094] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0095] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0096] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0097] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0098] In the above embodiments, the functions of each functional unit can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When loading and executing computer program instructions (programs) on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integrations. Available media may be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., high-density digital video discs (DVDs), or semiconductor media (e.g., solid state disks (SSDs), etc.).

[0099] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0100] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0101] In summary, the present application provides an intelligent drill evaluation method, system, medium, program product and terminal based on a big model and RAG, including: building a virtual training scenario based on a big language model; the trainer interacts with the virtual customer portrait in the virtual training scenario and collects the conversation interaction data in real time; uses RAG to analyze and feedback the virtual customer portrait and the conversation interaction data, and evaluates the comprehensive ability of the trainer based on multi-dimensional evaluation indicators.

[0102] The present invention not only helps trainers identify issues that virtual customers may be concerned about and provide answering strategies, but also comprehensively evaluates the professional capabilities of trainers from multiple dimensions such as answer fluency, objection problem handling, portrait question targeting, customer emotion processing, and insurance expertise to ensure the accuracy and fairness of the evaluation. The present invention helps to comprehensively improve the professional capabilities and service quality of trainers by combining virtual customer portrait analysis with insurance industry information and corporate private domain knowledge, and the multi-dimensional evaluation mechanism is of great significance for improving customer service quality and satisfaction. The virtual training scenario based on large models and RAG provided by the present invention can improve the effect of drills, improve customs clearance efficiency, and retain training data so that insurance agents can receive effective training in simulated conversations, thereby improving their insurance sales capabilities. Therefore, the present application effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0103] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. An intelligent drill evaluation method based on a large model and RAG, characterized in that: include: Build virtual training scenarios based on large language models; The trainer interacts with the virtual customer portrait in the virtual training scenario, and collects the conversation interaction data in real time; RAG is used to analyze and provide feedback on the virtual customer portrait and the conversation interaction data, and the comprehensive capabilities of the trainers are evaluated based on multi-dimensional evaluation indicators.

2. The intelligent drill evaluation method based on a large model and RAG according to claim 1 is characterized in that: The multi-dimensional evaluation indicators include: answer fluency, handling of objection questions, targeted nature of portrait questions, handling of customer emotions and insurance expertise.

3. The intelligent drill evaluation method based on large model and RAG according to claim 1 is characterized in that: The process of analyzing and feeding back the dialogue interaction data using RAG includes: performing vectorization processing on the conversation interaction data to obtain vectorized conversation interaction data; The vectorized conversation interaction data is queried and retrieved in a pre-built insurance industry information and enterprise private domain information knowledge base using RAG to obtain knowledge text that meets preset relevant requirements; Forming a prompt template based on the knowledge text and dialogue interaction data that meet the preset relevant requirements; Inputting the prompt template into the virtual training scene to generate a dialogue result corresponding to the dialogue interaction data; The dialogue result corresponding to the dialogue interaction data is fed back to the training personnel.

4. The intelligent drill evaluation method based on a large model and RAG according to claim 3 is characterized in that: The method further includes: after feeding back the dialogue result corresponding to the dialogue interaction data to the trainer, dynamically adjusting the dialogue strategy of the trainer to optimize the dialogue interaction data according to the dialogue strategy.

5. The intelligent drill evaluation method based on large model and RAG according to claim 3 is characterized in that: The construction process of the pre-built insurance industry information and enterprise private domain information knowledge base includes: Acquire insurance industry information and enterprise private domain knowledge, and process the insurance industry information and enterprise private domain knowledge to generate an initial knowledge base; Vectorizing the initial knowledge base using a pre-trained embedding model to obtain a vectorized initialization knowledge base; The vectorized initialization knowledge base is stored in a vector database to form an insurance industry information and enterprise private domain information knowledge base.

6. The intelligent drill evaluation method based on a large model and RAG according to claim 1 is characterized in that: The method further includes: evaluating the comprehensive ability of the trainer based on multi-dimensional evaluation indicators to obtain evaluation results, and visually displaying the evaluation results.

7. An intelligent drill evaluation system based on a large model and RAG, characterized in that: include: A virtual training scenario building module, used to build a virtual training scenario based on a large language model; A dialogue interaction data collection module is used for the training personnel to conduct dialogue interactions with the virtual customer portrait in the virtual training scenario and collect dialogue interaction data in real time; The comprehensive evaluation module is used to analyze and provide feedback on the virtual customer portrait and the conversation interaction data using RAG, and to evaluate the comprehensive capabilities of the trainers based on multi-dimensional evaluation indicators.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the intelligent rehearsal evaluation method based on a large model and RAG as described in any one of claims 1 to 6 is implemented.

9. A computer program product, characterized in that The computer program product includes computer program code, and when the computer program code is executed on a computer, the computer is enabled to implement the intelligent exercise evaluation method based on a large model and RAG as described in any one of claims 1 to 6.

10. An electronic terminal comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the intelligent exercise evaluation method based on a large model and RAG as described in any one of claims 1 to 6.

Images