Multi-role training system and method based on multi-modal large model

Abstract

The invention provides a multi-role training system and method based on a multi-modal large model, and the system comprises a training knowledge base generation module which is used for generating a training knowledge base; the role setting module is used for setting roles according to the personal information of the trainees; the interaction module is used for carrying out questioning and real-time evaluation according to the selected role and the training knowledge base; and the evaluation module is used for generating final evaluation. According to the technical scheme, the training role can be customized according to the self condition of the trainee, customers possibly encountered by the trainee in a real scene can be imitated, and the learning effect of the trainee in training is improved; during interaction, the trainees can select interaction modes including voice interaction and text interaction, so that the training process is more flexible, and answers of the trainees are fed back in real time.

Description

Background Art

[0002] Training is an important part of corporate management. Online training has become a common training method, with advantages such as flexibility, cost-effectiveness and diversified forms. However, there have always been problems in the interaction between instructors and trainees, especially in the question-and-answer session. Although the current AI-based question-and-answer robots can provide training in professional knowledge, they cannot simulate real-life scenarios when training trainees. The training roles are single, and most of them only set up the examiner role for training.

[0003] The prior art (invention patent with announcement number CN117149984B) discloses a customized training method and device based on a large model thinking chain, which is applied to the field of training information technology. Through customized training information, it solves the problem that the instructor's manual response in the question-and-answer session is inefficient and the robot question-and-answer lacks professionalism; however, in actual situations, first, different training scenarios will involve different training forms. For example, in the scenario of training sales personnel, multiple roles need to be set for customized training; second, trainees need to interact in real time and flexibly. Trainees can get real-time feedback and can freely choose text interaction or voice interaction. Therefore, a multi-role training system based on a multimodal large model is needed. Summary of the invention

[0004] The present application provides a multi-role training system and method based on a multimodal large model to improve the learning effect of trainees in training.

[0005] In a first aspect, a multi-role training system based on a multimodal large model is provided, comprising:

[0006] A training knowledge base generation module, used for generating a training knowledge base;

[0007] The role setting module is used to set roles according to the students’ personal information;

[0008] Interactive module for asking questions and real-time evaluation based on the selected role and training knowledge base;

[0009] Evaluation module, used to generate the final evaluation.

[0010] In the above technical solution, a training knowledge base generation module is used to generate a training knowledge base; a role setting module is used to set roles according to the trainees' personal information; an interaction module is used to ask questions and make real-time evaluations based on the selected roles and training knowledge base; an evaluation module is used to generate a final evaluation; the training roles can be customized according to the trainees' own circumstances, and the customers that the trainees may encounter in real scenarios can be simulated to improve the trainees' learning effects in training; during interaction, the trainees can choose the interaction method, including voice interaction and text interaction, to make the training process more flexible, and provide real-time feedback on the trainees' answers; after the training, the trainees' answers are evaluated as a whole, and a comprehensive evaluation report is issued.

[0011] In a specific implementation scheme, the training knowledge base generation module includes:

[0012] A text extraction unit, used for extracting document content into text;

[0013] A preprocessing unit, used for performing preprocessing;

[0014] The knowledge base unit is used to generate question-answer pairs using a large model and form structured data to generate a knowledge base.

[0015] In a specific implementation scheme, the role setting module includes:

[0016] The user portrait unit is used to obtain the trainee information and the customer information corresponding to the trainee and generate the user portrait of the trainee;

[0017] The role unit is used to use the large model to generate different training roles according to the corresponding customer information and screen the valid roles.

[0018] In a specific implementation scheme, the interaction module includes:

[0019] Selection unit, used to select roles and training content;

[0020] A formulation unit is used to formulate different interaction strategies according to different roles;

[0021] The question extraction unit is used to extract setting questions of different roles.

[0022] In a specific implementation scheme, the evaluation module includes:

[0023] An evaluation dimension unit, used to set the evaluation dimension;

[0024] The evaluation report unit is used to generate an evaluation report.

[0025] In a second aspect, a multi-role training method based on a multimodal large model is provided, comprising the following steps:

[0026] Generate a training knowledge base using a training knowledge base generation module;

[0027] Use the role setting module to set roles according to students’ personal information;

[0028] Use interactive modules to ask questions and conduct real-time evaluation based on the selected role and training knowledge base;

[0029] Use the assessment module to generate the final evaluation.

[0030] In the above technical solution, a training knowledge base generation module is used to generate a training knowledge base; a role setting module is used to set roles according to the trainees' personal information; an interaction module is used to ask questions and make real-time evaluations based on the selected roles and training knowledge base; an evaluation module is used to generate a final evaluation; the training roles can be customized according to the trainees' own circumstances, and the customers that the trainees may encounter in real scenarios can be simulated to improve the trainees' learning effects in training; during interaction, the trainees can choose the interaction method, including voice interaction and text interaction, to make the training process more flexible, and provide real-time feedback on the trainees' answers; after the training, the trainees' answers are evaluated as a whole, and a comprehensive evaluation report is issued.

[0031] In a specific implementation scheme, the training knowledge base generation module includes:

[0032] A text extraction unit, used for extracting document content into text;

[0033] A preprocessing unit, used for performing preprocessing;

[0034] The knowledge base unit is used to generate question-answer pairs using a large model and form structured data to generate a knowledge base.

[0035] In a specific implementation scheme, the role setting module includes:

[0036] The user portrait unit is used to obtain the trainee information and the customer information corresponding to the trainee and generate the user portrait of the trainee;

[0037] The role unit is used to use the large model to generate different training roles according to the corresponding customer information and screen the valid roles.

[0038] In a specific implementation scheme, the interaction module includes:

[0039] Selection unit, used to select roles and training content;

[0040] A formulation unit is used to formulate different interaction strategies according to different roles;

[0041] The question extraction unit is used to extract setting questions of different roles.

[0042] In a specific implementation scheme, the evaluation module includes:

[0043] An evaluation dimension unit, used to set the evaluation dimension;

[0044] The evaluation report unit is used to generate an evaluation report. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A structural block diagram of a multi-role training system based on a multi-modal large model provided in an embodiment of the present application;

[0046] Figure 2 A flowchart of a multi-role training method based on a multi-modal large model provided in an embodiment of the present application;

[0047] Figure 3 A specific flow chart of the multi-role training method based on a multimodal large model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The present application is further described in detail below through the accompanying drawings and embodiments. Through these descriptions, the characteristics and advantages of the present application will become clearer and more specific.

[0049] The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise noted.

[0050] In addition, the technical features involved in different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0051] To facilitate understanding of the multi-role training system and method based on a multimodal large model provided in the embodiment of the present application, its application scenario is first explained. The multi-role training system and method based on a multimodal large model provided in the embodiment of the present application are used to improve the learning effect of trainees in training. Training is an important part of enterprise management. Online training has become a common training method, which has the advantages of flexibility, cost-effectiveness and diversified forms. However, there have always been problems in the interaction between instructors and trainees, especially in the question-and-answer session. Although the current question-and-answer robot based on artificial intelligence can be trained in professional knowledge, it is impossible to imitate the scene in the real situation for training when training trainees. The training role is single, and most of them only set the examiner role for training. The prior art (invention patent with announcement number CN117149984B) discloses a customized training method and device based on a large model thinking chain, which is applied to the field of training information technology. Through customized training information, the problem of low efficiency of manual responses by instructors in the question-and-answer session and lack of professionalism in robot question-and-answer is solved; however, in actual situations, first, different training scenarios will involve different training forms. For example, in the scenario of training sales personnel, it is necessary to set up multiple roles for customized training; second, trainees need to interact in real time and flexibly. Trainees can get real-time feedback and can freely choose text interaction or voice interaction. Therefore, a multi-role training system based on a multimodal large model is needed. For this purpose, the embodiment of the present application provides a multi-role training system and method based on a multimodal large model to improve the learning effect of trainees in training. The following is a detailed description of the embodiment in conjunction with specific drawings.

[0052] refer to Figures 1 to 3 , Figure 1 A structural block diagram of a multi-role training system based on a multi-modal large model provided in an embodiment of the present application; Figure 2 A flowchart of a multi-role training method based on a multi-modal large model provided in an embodiment of the present application; Figure 3 A specific flow chart of the multi-role training method based on a multimodal large model provided in an embodiment of the present application.

[0053] exist Figure 1 In the embodiment of the present application, a multi-role training system based on a multi-modal large model is provided, comprising:

[0054] A training knowledge base generation module, used for generating a training knowledge base;

[0055] The role setting module is used to set roles according to the students’ personal information;

[0056] Interactive module for asking questions and real-time evaluation based on the selected role and training knowledge base;

[0057] Evaluation module, used to generate the final evaluation.

[0058] In the above technical solution, a training knowledge base generation module is used to generate a training knowledge base; a role setting module is used to set roles according to the trainees' personal information; an interaction module is used to ask questions and make real-time evaluations based on the selected roles and training knowledge base; an evaluation module is used to generate a final evaluation; the training roles can be customized according to the trainees' own circumstances, and the customers that the trainees may encounter in real scenarios can be simulated to improve the trainees' learning effects in training; during interaction, the trainees can choose the interaction method, including voice interaction and text interaction, to make the training process more flexible, and provide real-time feedback on the trainees' answers; after the training, the trainees' answers are evaluated as a whole, and a comprehensive evaluation report is issued.

[0059] In a specific implementation scheme, the training knowledge base generation module includes:

[0060] A text extraction unit, used for extracting document content into text;

[0061] A preprocessing unit, used for performing preprocessing;

[0062] The knowledge base unit is used to generate question-answer pairs using a large model and form structured data to generate a knowledge base.

[0063] In a specific implementation scheme, the role setting module includes:

[0064] The user portrait unit is used to obtain the trainee information and the customer information corresponding to the trainee and generate the user portrait of the trainee;

[0065] The role unit is used to use the large model to generate different training roles according to the corresponding customer information and screen the valid roles.

[0066] In a specific implementation scheme, the interaction module includes:

[0067] Selection unit, used to select roles and training content;

[0068] A formulation unit is used to formulate different interaction strategies according to different roles;

[0069] The question extraction unit is used to extract setting questions of different roles.

[0070] In a specific implementation scheme, the evaluation module includes:

[0071] An evaluation dimension unit, used to set the evaluation dimension;

[0072] The evaluation report unit is used to generate an evaluation report.

[0073] exist Figure 2In the embodiment of the present application, a multi-role training method based on a multimodal large model is provided, comprising the following steps:

[0074] Generate a training knowledge base using a training knowledge base generation module;

[0075] Use the role setting module to set roles according to students’ personal information;

[0076] Use interactive modules to ask questions and conduct real-time evaluation based on the selected role and training knowledge base;

[0077] Use the assessment module to generate the final evaluation.

[0078] In the above technical solution, a training knowledge base generation module is used to generate a training knowledge base; a role setting module is used to set roles according to the trainees' personal information; an interaction module is used to ask questions and make real-time evaluations based on the selected roles and training knowledge base; an evaluation module is used to generate a final evaluation; the training roles can be customized according to the trainees' own circumstances, and the customers that the trainees may encounter in real scenarios can be simulated to improve the trainees' learning effects in training; during interaction, the trainees can choose the interaction method, including voice interaction and text interaction, to make the training process more flexible, and provide real-time feedback on the trainees' answers; after the training, the trainees' answers are evaluated as a whole, and a comprehensive evaluation report is issued.

[0079] In a specific implementation scheme, the training knowledge base generation module includes:

[0080] A text extraction unit, used for extracting document content into text;

[0081] A preprocessing unit, used for performing preprocessing;

[0082] The knowledge base unit is used to generate question-answer pairs using a large model and form structured data to generate a knowledge base.

[0083] In a specific implementation scheme, the role setting module includes:

[0084] The user portrait unit is used to obtain the trainee information and the customer information corresponding to the trainee and generate the user portrait of the trainee;

[0085] The role unit is used to use the large model to generate different training roles according to the corresponding customer information and screen the valid roles.

[0086] In a specific implementation scheme, the interaction module includes:

[0087] Selection unit, used to select roles and training content;

[0088] A formulation unit is used to formulate different interaction strategies according to different roles;

[0089] The question extraction unit is used to extract setting questions of different roles.

[0090] In a specific implementation scheme, the evaluation module includes:

[0091] An evaluation dimension unit, used to set the evaluation dimension;

[0092] The evaluation report unit is used to generate an evaluation report.

[0093] Specifically, refer to Figure 3 The multi-role training method based on a multi-modal large model is a multi-role training system based on a multi-modal large model, which trains trainees by setting two main roles, examiner and customer, and the interaction mode includes text interaction and voice interaction, and specifically includes the following steps:

[0094] S1: The training knowledge generation module generates a training knowledge base based on the training materials. The specific steps are as follows:

[0095] Step 11: Extract the document content in any format into text for model processing;

[0096] Step 12: Perform preprocessing, such as removing spaces, special characters, etc., and breaking the text into blocks suitable for model processing;

[0097] Step 13: Use the big model to generate question-answer pairs and form structured data. The question-answer pairs are divided into different modules to form a knowledge base.

[0098] Specifically, the large model adopts a multimodal large language model.

[0099] Specifically, the Multimodal Large Language Model (MLLM) is a technology that combines vision and large language models.

[0100] The multimodal large language model can perceive inputs of different modes such as pictures and texts, and learn the context and generate answers in an autoregressive manner based on the instructions given by humans. It relies on the rich knowledge reserve and strong reasoning and generalization capabilities of the large language model (LLM) to solve multimodal problems, such as writing with pictures and writing code with pictures. The multimodal large language model uses technologies such as natural language processing, computer vision, and speech recognition, and integrates them into a system, so that it can understand human language and emotions more accurately.

[0101] A typical multimodal large language model can be abstracted into three modules, namely the pre-trained modality encoder, the pre-trained large language model (LLM), and the modality interface connecting them. Analogously to humans, modality encoders such as image / audio encoders are like human eyes / ears that receive and pre-process optical / acoustic signals, while LLMs are like human brains that understand and reason about the processed signals. In between, the modality interface is used to align different modalities. Some MLLMs also include a generator to output other modalities besides text.

[0102] Modality Encoder: An encoder compresses raw information, such as images or audio, into a more compact representation. Instead of training from scratch, a common approach is to use a pre-trained encoder that has been aligned with the other modality.

[0103] Pre-trained LLM: Instead of training an LLM from scratch, a more efficient and practical approach is to start from a pre-trained model. By pre-training on a large number of web corpora, LLMs have embedded rich world knowledge and demonstrated strong generalization and reasoning capabilities.

[0104] The training of a large multimodal language model usually includes two stages: multimodal pre-training and multimodal instruction tuning.

[0105] Multimodal pre-training: Using the X-Text dataset, we train the input and output projectors by optimizing predefined objectives to achieve alignment between various modalities.

[0106] Multimodal Instruction Fine-tuning: A method for fine-tuning pre-trained multimodal large language models using instruction-formatted datasets. Through this process, the model can generalize to unseen tasks by following new instructions, thereby improving zero-shot performance.

[0107] Advantages include: Multimodal Contextual Learning (M-ICL): An effective technique commonly used in the inference phase to improve few-shot performance.

[0108] Multimodal Chained Reasoning (M-CoT): Usually used for complex reasoning tasks.

[0109] LLM-based Visual Reasoning (LAVR): for joint reasoning tasks across visual and language modalities.

[0110] The multimodal large language model makes the system more intelligent and humane by integrating information from multiple modalities such as text, voice, and images.

[0111] S2: The role setting module is to set the role according to the student's personal information. The specific steps are as follows:

[0112] Step 21: Obtain the trainee information and the customer information corresponding to the trainee, and generate the user portrait of the trainee;

[0113] Step 22: Use the big model to generate different training roles based on the corresponding information and screen the effective roles.

[0114] S3: The interactive module asks questions and conducts real-time evaluation based on the selected role and training knowledge base. The specific steps are as follows:

[0115] Step 31: Trainees choose roles and training content. Roles are divided into coaches and clients. There are many types of clients, and the training content involves different training knowledge.

[0116] Step 32: Develop different interaction strategies based on different roles;

[0117] Step 33: If the coach role is selected, first, randomly select questions from each module of the knowledge base; second, determine whether the student's answer is related to the question raised. If so, evaluate the answer based on the answer matching the question raised and raise the next round of questions. The evaluation criteria are correctness and clarity of expression. If not (including interruption, etc.), reply to the student and raise the previous round of questions;

[0118] If you choose the customer role, first, randomly select questions and ask them in combination with the customer persona; second, determine whether the college's answer has answered the customer role's questions. If it has, ask the next round of questions based on the persona; if it has not, continue to ask the previous round of questions;

[0119] Step 34: After the examiner and the customer roles have respectively extracted all the set questions, the interactive module ends.

[0120] S4: The evaluation module generates the final evaluation based on the chat records and real-time evaluation. The specific steps are as follows:

[0121] Step 41: Set evaluation dimensions, including accuracy, clarity of expression, politeness, communication skills, etc.;

[0122] Step 42: Generate an evaluation report based on the chat history and real-time evaluation.

[0123] In the above technical solution, the training roles are not limited to examiners. Different training roles can be generated according to the trainee information to simulate customers in real scenarios. The real-time evaluation is combined with the overall evaluation. The training roles can be customized according to the trainees' own situations, and customers that trainees may encounter in real scenarios can be imitated to improve the trainees' learning effect in training. When interacting, trainees can choose the interaction method, including voice interaction and text interaction, which makes the training process more flexible and provides real-time feedback on the trainees' answers. After the training, the trainees' answers are evaluated as a whole and a comprehensive evaluation report is issued.

[0124] Those skilled in the art will appreciate that the present application may be implemented as a system, method or computer program product.

[0125] Therefore, the present disclosure may be specifically implemented in the following forms, namely: it may be completely hardware, it may be completely software (including firmware, resident software, microcode, etc.), or it may be a combination of hardware and software, generally referred to herein as a "circuit", "module" or "system". In addition, in some embodiments, the present application may also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium may contain computer-readable program code.

[0126] Any combination of one or more computer-readable media can be used. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0127] Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art can change, modify, replace and modify the above embodiments within the scope of the present application. On this basis, a variety of replacements and improvements can be made to the present application, all of which fall within the scope of protection of the present application.

Claims

1. A multi-role training system based on a multi-modal large model, characterized in that: include: A training knowledge base generation module, used for generating a training knowledge base; The role setting module is used to set roles according to the students’ personal information; Interactive module for asking questions and real-time evaluation based on the selected role and training knowledge base; Evaluation module, used to generate the final evaluation.

2. The multi-role training system based on a multi-modal large model according to claim 1 is characterized in that: The training knowledge base generation module includes: A text extraction unit, used for extracting document content into text; A preprocessing unit, used for performing preprocessing; The knowledge base unit is used to generate question-answer pairs using a large model and form structured data to generate a knowledge base.

3. The multi-role training system based on a multi-modal large model according to claim 2 is characterized in that: The role setting module includes: The user portrait unit is used to obtain the trainee information and the customer information corresponding to the trainee and generate the user portrait of the trainee; The role unit is used to use the large model to generate different training roles according to the corresponding customer information and screen the valid roles.

4. The multi-role training system based on a multi-modal large model according to claim 3 is characterized in that: The interaction module includes: Selection unit, used to select roles and training content; A formulation unit is used to formulate different interaction strategies according to different roles; The question extraction unit is used to extract setting questions of different roles.

5. The multi-role training system based on a multi-modal large model according to claim 4 is characterized in that: The evaluation module includes: An evaluation dimension unit, used to set the evaluation dimension; The evaluation report unit is used to generate an evaluation report.

6. A multi-role training method based on a multi-modal large model, characterized in that: The following steps are involved: Generate a training knowledge base using a training knowledge base generation module; Use the role setting module to set roles according to students’ personal information; Use interactive modules to ask questions and conduct real-time evaluation based on the selected role and training knowledge base; Use the assessment module to generate the final evaluation.

7. The multi-role training method based on a multi-modal large model according to claim 6 is characterized in that: The training knowledge base generation module includes: A text extraction unit, used for extracting document content into text; A preprocessing unit, used for performing preprocessing; The knowledge base unit is used to generate question-answer pairs using a large model and form structured data to generate a knowledge base.

8. The multi-role training method based on a multi-modal large model according to claim 7 is characterized in that: The role setting module includes: The user portrait unit is used to obtain the trainee information and the customer information corresponding to the trainee and generate the user portrait of the trainee; The role unit is used to use the large model to generate different training roles according to the corresponding customer information and screen the valid roles.

9. The multi-role training method based on a multi-modal large model according to claim 8 is characterized in that: The interaction module includes: Selection unit, used to select roles and training content; A formulation unit is used to formulate different interaction strategies according to different roles; The question extraction unit is used to extract setting questions of different roles.

10. The multi-role training method based on a multi-modal large model according to claim 9 is characterized in that: The evaluation module includes: An evaluation dimension unit, used to set the evaluation dimension; The evaluation report unit is used to generate an evaluation report.

Images