An intelligent training system based on an end-to-end multimodal teaching model

Abstract

The present application discloses an intelligent training system based on an end-to-end multimodal teaching large model, which relates to the field of training information technology, and solves the technical problem that the existing artificial intelligence training is difficult to answer the questions raised by the trainees in time, resulting in poor effect of intelligent training; it includes an interactive module: extracting training documents, personalized modification information, question information and test answers in the training materials; a training material generation module: generating training materials according to the training documents and personalized modification information; a question and answer module: generating answer information according to the question information; an evaluation module: generating an evaluation test paper according to the training materials and question information; generating a training evaluation according to the evaluation test paper; generating personalized training materials according to the training evaluation of the evaluation test paper; a training module: conducting automatic voice explanation training according to the training materials; answering according to the answer information generated by the question information; and generating personalized training materials according to the training evaluation of the evaluation test paper; and improving the training effect of intelligent training.

Description

Technical Field

[0001] The present application belongs to the field of training information technology, and relates to training information technology, specifically an intelligent training system based on an end-to-end multimodal teaching large model. Background Art

[0002] Training is an important part of enterprise management. Online training has become a common training method. Online training has great advantages in that it is flexible in time and place. However, there have always been problems in the interaction between instructors and trainees, especially in the question-and-answer session. With the development of artificial intelligence technology, the use of conversational robots can provide certain help on the basis of traditional manual question-and-answer sessions, thus reducing training costs to a certain extent.

[0003] The prior art (invention patent with announcement number CN117149984B) discloses a customized training method and device based on a large model thinking chain, which is applied to the field of training information technology, including: obtaining text materials for training, establishing a training knowledge base, and extracting keywords from the training knowledge base, obtaining a list of historical questions of all trainees and current questions of target trainees, obtaining an input prompt sequence with a length of a preset number of times, and calculating the possibility of each input prompt, inputting the most likely input prompt into a pre-trained large model, and obtaining customized training information. The present invention is aimed at the problem that the manual response of teachers in the question-and-answer session during online training is inefficient and the robot question-and-answer lacks professionalism. Based on the large model thinking chain, a method for generating customized training materials is provided, which can provide high-quality customized training materials.

[0004] The above-mentioned customized training method based on the big model thinking chain solves the problem of low efficiency of instructors' manual responses in the question-and-answer session and lack of professionalism of robot questions and answers by customizing training information; however, in the actual training process, the trainees may have questions about any knowledge point in the intelligent training process at any time. If the questions cannot be answered in time, the corresponding trainees will have poor learning effects in subsequent training; when training on knowledge points that are more difficult and have strong coherence, the effect of intelligent training will be poor. Therefore, an intelligent training system based on an end-to-end multimodal teaching big model is needed. Summary of the invention

[0005] The present application aims to solve at least one of the technical problems existing in the prior art; to this end, the present application proposes an intelligent training system based on an end-to-end multimodal teaching large model, which is used to solve the technical problem that the existing artificial intelligence training is difficult to answer questions raised by the trainees in a timely manner, resulting in poor results of intelligent training.

[0006] To achieve the above-mentioned purpose, the first aspect of the present application provides an intelligent training system based on an end-to-end multimodal teaching model, comprising: a training material generation module, a question-and-answer module, an evaluation module, a training module, and an interaction module;

[0007] The interactive module is used to extract training documents, personalized modification information, question information and test answers from the training materials;

[0008] The training material generation module generates training materials according to the training documents and personalized modification information; the training materials include training documents, explanation documents and teaching voices; the training documents include PPT outlines and training content;

[0009] The question-answering module generates answer information according to the question information;

[0010] The evaluation module: generates an evaluation test paper based on the training materials and question information; generates a training evaluation based on the evaluation test paper;

[0011] The training module: conducts training according to the training materials; provides answers according to the answer information generated by the question information; and generates personalized training materials according to the training evaluation of the assessment test paper, and sends the personalized training materials to the corresponding users.

[0012] This application realizes the whole process of intelligent training by generating training materials based on training documents and personalized modification information; conducting training based on training materials; obtaining trainees' questions during training; generating answer information based on question information; answering questions based on answer information generated from question information; generating assessment papers based on training materials and question information; generating training evaluation based on paper answers; generating personalized training materials based on training evaluation of assessment papers; ensuring trainees' learning effect in this training by answering trainees' questions in a timely manner, thereby increasing the training effect of this training.

[0013] Preferably, generating training materials according to the training documents and the personalized modification information comprises the following steps:

[0014] Step 1: Extract training documents from training materials, pre-process the training documents to obtain key content, and input the key content into the large language model to obtain the PPT outline;

[0015] Step 2: Obtain the user's personalized modification information for the PPT outline; determine whether the personalized modification information is "none", if yes, mark the PPT outline as a confirmed PPT outline; proceed to step 3; if no, modify the PPT outline according to the personalized modification information, and proceed to step 3;

[0016] Step 3: Input the confirmed PPT outline into the large language model to obtain training content, wherein the training content includes training text and training pictures; generate a training PPT according to the PPT outline, training text and training pictures;

[0017] Step 4: Obtain the training text and training pictures of each page in the training content; input them into the large language model to obtain the corresponding explanation document;

[0018] Step 5: Use Text to Speech technology to generate the explanation document into teaching voice;

[0019] Step 6: Integrate the training PPT, as well as the explanation documents and teaching voices corresponding to each training content in the training PPT into training materials.

[0020] Preferably, generating answer information according to the question information comprises the following steps:

[0021] Step 1: extracting the question voice in the question information, and converting the question voice into question text;

[0022] Step 2: preprocessing the question text to obtain a clean question text; the preprocessing includes character filtering and noise processing;

[0023] Step 3: Vectorize the question text to obtain the query vector; then use the query vector to search the document library to obtain the most similar document vector;

[0024] Step 4: Generate answer information based on the query vector and document vector.

[0025] Preferably, generating answer information according to the query vector and the document vector includes:

[0026] Step 1: Input the query vector and the document vector into the answer generation model to obtain the initial answer text; the answer generation model includes the RAG generation model;

[0027] Step 2: Check and revise the grammar of the initial answer text;

[0028] Step 3: Obtain the semantic relevance between the initial answer text and the question text; when the semantic relevance is greater than the set threshold, mark the initial answer text as the answer text; proceed to step 4; otherwise, proceed to step 1;

[0029] Step 4: Generate answer voice according to the answer text; integrate the answer text and answer voice into answer information.

[0030] Preferably, answering the question based on the answer information generated by the question information includes:

[0031] Step 1: After the answer information is generated, obtain the question text and extract the answer text and answer voice in the answer information;

[0032] Step 2: Get the current training moment, and determine whether the current training moment is a speech node, where the speech node is the time node between the corresponding moment when each sentence in the teaching speech ends and the beginning of the next sentence; if yes, proceed to step 3; if no, wait for the arrival of the speech node; when the speech node arrives, proceed to step 3;

[0033] Step 3: Mark the speech node as an interruption node; display the question text and the answer text, and play the answer voice; and obtain the partial explanation document before the speech node, and generate a transition explanation document and a transition teaching voice according to the partial explanation document; when the answer voice ends, proceed to step 4;

[0034] Step 4: Play the transition teaching voice; after the teaching voice is finished playing, jump to the interruption node and continue playing the teaching voice.

[0035] Preferably, the generating of the transition explanation document and the transition teaching voice according to the partial explanation document comprises:

[0036] Obtain partial explanation documents and summary vocabulary ratio ranges; input partial explanation documents and summary vocabulary ratio ranges into the large language model to obtain transition explanation documents, and generate transition teaching voices according to the transition explanation documents.

[0037] Preferably, generating an assessment test paper according to the training materials and question information comprises the following steps:

[0038] Step 1: Extract question information and knowledge points from each training document in the training materials

[0039] Step 2: Obtain training objectives, and generate test questions based on the training objectives and knowledge points; the test questions include single-choice questions, multiple-choice questions, and short-answer questions;

[0040] Step 3: Integrate several test questions into an assessment paper.

[0041] Preferably, generating a training evaluation based on the assessment test paper includes:

[0042] The test answers of each student are obtained, and the test answers are input into the intelligent scoring model to obtain the corresponding test scores, and the training evaluation is integrated according to the test scores.

[0043] Preferably, generating personalized training materials according to the training evaluation of the test paper comprises the following steps:

[0044] Step 1: Obtain the professional vocabulary proportion ZB, general vocabulary proportion PB and auxiliary vocabulary proportion FB of this explanation document; as well as the test score YPi of each student in the training evaluation, where i is the number of each student;

[0045] Step 2: Determine whether the test score YPi of each student is greater than the set training evaluation standard BP; if yes, proceed to step 3; if no, proceed to step 4;

[0046] Step 3: Calculate the adjustment ratio TBi corresponding to the student numbered i by the formula TBi=(YPi-BP) / MP×KB1; calculate the personalized professional vocabulary ratio GZB by the formula GZB=ZB+TBi; calculate the personalized general vocabulary ratio GPB by the formula GPB=PB-α1×TBi; calculate the personalized auxiliary vocabulary ratio GFB by the formula GFB=FB-α2×TBi; where α1 and α2 are proportional coefficients; and α1+α2=1; jump to step 5;

[0047] Step 4: Calculate the adjustment ratio TBi corresponding to the student numbered i by the formula TBi=(BP-YPi) / MP×KB2; calculate the personalized professional vocabulary ratio GZB by the formula GZB=ZB-TBi; calculate the personalized general vocabulary ratio GPB by the formula GPB=PB+α1×TBi; calculate the personalized auxiliary vocabulary ratio GFB by the formula GFB=FB+α2×TBi; where α1 and α2 are proportional coefficients; and α1+α2=1; proceed to step 5;

[0048] Step 5: Generate personalized explanation documents and personalized teaching voices according to the proportion of personalized professional vocabulary, personalized general vocabulary, personalized auxiliary vocabulary, and corresponding training content; package the personalized explanation documents, personalized teaching voices, and training content into personalized training materials, and send them to corresponding students.

[0049] Compared with the prior art, the beneficial effects of this application are:

[0050] 1. This application realizes the whole process of intelligent training by generating training materials according to training documents and personalized modification information; conducting training according to the training materials; obtaining trainees' questions during the training; generating answer information according to the question information; answering questions according to the answer information generated from the question information; generating evaluation papers according to the training materials and question information; generating training evaluation according to the answers to the test papers; generating personalized training materials according to the training evaluation of the evaluation papers; and ensuring the learning effect of the trainees in this training by answering the trainees' questions in a timely manner, thereby increasing the training effect of this training.

[0051] 2. This application provides students with personalized training materials through the analysis results of each student's test scores to help students improve their knowledge and skills in a targeted manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 This is a schematic diagram of the principle of the intelligent training system in this application;

[0054] Figure 2 Generate step diagrams for the training materials in this application;

[0055] Figure 3 Generate a flow chart for answering information in this application. DETAILED DESCRIPTION

[0056] The technical solution of the present application will be described clearly and completely below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0057] See also Figure 1 , the first aspect of the present application provides an intelligent training system based on an end-to-end multimodal teaching model, including: a training material generation module, a question and answer module, an evaluation module, a training module and an interaction module;

[0058] Interactive module: used to extract training documents, personalized modification information, question information and test answers from training materials; training documents are the general content and materials of this training; personalized modification information is the user's modification suggestions for the generated PPT outline, etc.; question information is the voice or text of the trainees' questions obtained in real time during the training process; test answers are the answers of each trainee to the assessment test paper;

[0059] Training material generation module: generates training materials based on training documents and personalized modification information; training materials include training documents, explanation documents and teaching voices; the training documents include PPT outlines and training content; the training content includes training texts and training pictures, the training text is the content of each page generated based on the PPT outline, and the training pictures are the illustrations of each page generated based on the PPT outline and training text; the explanation document is the explanation content generated based on the training content, and the teaching voice is the output voice converted from the explanation content.

[0060] Question and answer module: generates answer information based on question information; the answer information is the relevant answer generated based on the question information;

[0061] Evaluation module: Generates evaluation papers based on training materials and question information; generates training evaluation based on test paper answers; the evaluation papers are the papers generated based on this training material, and the training evaluation is the answers of each student to this evaluation paper;

[0062] Training module: conduct training according to training materials; answer questions based on answer information generated from question information; and generate personalized training materials based on the training evaluation of the assessment paper, and send the personalized training materials to the corresponding users; personalized training materials are relevant materials for this training generated based on the learning situation of each student.

[0063] This embodiment realizes the whole process of intelligent training by generating training materials based on training documents and personalized modification information; conducting training based on the training materials; obtaining trainees' question information during the training; generating answer information based on the question information; answering based on the answer information generated from the question information; generating evaluation papers based on the training materials and question information; generating training evaluation based on the answers to the papers; generating personalized training materials based on the training evaluation of the evaluation papers; and ensuring the learning effect of the trainees in this training by promptly answering the trainees' question information, thereby increasing the training effect of this training.

[0064] See also Figure 2 , generating training materials based on training documents and personalized modification information, including the following steps:

[0065] Step 1: Extract training documents from training materials, pre-process the training documents to obtain key content, and input the key content into the large language model to obtain the PPT outline; the pre-processing process is as follows: document format recognition: including but not limited to Word documents, PDF files, etc.; automatically identify the format of uploaded documents and extract text content; document content extraction is to extract text content from training documents using natural language processing technology; after extracting the document content, it is also necessary to remove irrelevant information, mark titles and paragraphs, etc.;

[0066] It is understandable that users can choose appropriate PPT templates according to actual needs and personal preferences to meet presentation needs in different scenarios.

[0067] The large language model here is a large language model obtained by training the original large language model through several key contents and PPT outlines; the large language model can also be guided to generate a PPT outline that meets user needs by designing prompt words (Prompt Engineering). The design of prompt words is intended to clarify the requirements for model-generated content, including the structure of the outline, the focus of the content and the theme of the presentation, so as to generate high-quality PPT outlines and presentation materials; the format and content of the prompt words in this embodiment are "You are a professional PPT outline generator. You need to generate a PPT outline based on the uploaded document content. The outline should be generated according to the idea of ​​general-specific-general. You need to return a PPT outline in json format. The json format is as follows {"SummaryHeading": "General Title", "Sections": [{"SectionHeader": "Section Title 1", "content": ["Content 1", "1. Content 1\\n2. Content 2\\n3. Content 3"]}, {"SectionHeader": "Section Title 2", "content": ["Content 4"]}]}

[0068] Note: The content of an element in a content corresponds to the theme content of a slide.

[0069] Step 2: Obtain the personalized modification information of the user on the PPT outline; the personalized modification information is the content of the modification of the PPT outline input by the user through the interactive module; if no modification is required, the corresponding personalized modification information content is "none"; determine whether the personalized modification information is "none", if yes, mark the PPT outline as a confirmed PPT outline; proceed to step 3; if no, modify the PPT outline according to the personalized modification information, and proceed to step 3;

[0070] Step 3: Input the confirmed PPT outline into the large language model to obtain training content, wherein the training content includes training text and training pictures; generate a training PPT according to the PPT outline, training text and training pictures;

[0071] The large language model here is a large language model obtained by training the original large language model through several confirmed PPT outlines and their corresponding training contents; the outline content corresponding to each page of the PPT is input into the large language model to obtain its corresponding training text and training pictures;

[0072] In this embodiment, the corresponding training text is generated by Generated_Content=h(Structured_Outline), wherein Structured_Outline is the outline content; h() is the corresponding content generation function; Generated_Content is the training text; and the training image is generated by the Stable Diffusion model. Specifically, the training text is input into the Stable Diffusion model, and the Stable Diffusion model generates a training image according to the training text.

[0073] According to the template selected by the user, the system automatically matches the appropriate content layout. The system considers the design style and layout requirements of the template, organizes the extracted document content into PPT pages that conform to the template format, arranges the text content, images and other design elements according to the template requirements, and generates a complete PPT file.

[0074] Step 4: Obtain the training text and training pictures of each page in the training content; input them into the large language model to obtain the corresponding explanation document;

[0075] First, convert the information in the PPT into text format, and use the large language model to perform content analysis on each slide. This process includes identifying key information, chart data, and core concepts on the page to ensure a comprehensive understanding of the content. This includes: Explanation of the main information points: Based on the results of the content analysis, the model generates a detailed content introduction for each slide. The explanatory text includes a detailed explanation of the identified key information to ensure that the content is structured, logically clear, and closely related to the training objectives.

[0076] Explanation of chart data: For slides containing charts, the model generates detailed explanations of the chart data. The explanation includes the source, meaning and importance of the data, and uses simple and clear language to explain complex data to help employees better understand and absorb the information.

[0077] The large language model here is obtained by training several different training contents and several corresponding explanation documents with different vocabulary proportions based on the large language model; the vocabulary proportion is the proportion of different vocabulary in the explanation document, including professional vocabulary proportion, common vocabulary proportion and auxiliary vocabulary proportion; it can be understood that when inputting the training text and training pictures corresponding to each page of the slide, a vocabulary proportion range is set by default inside the large language model, and the corresponding vocabulary proportion of the generated explanation document is within the set vocabulary proportion range. The vocabulary proportion range in this embodiment is set to a professional vocabulary proportion range of 25%-30%, a common vocabulary proportion of 65%-70%, and an auxiliary vocabulary proportion of 5%-10%; the specific proportion range is set according to experience.

[0078] Step 5: Use Text to Speech technology to generate the explanation document into teaching voice;

[0079] Step five specifically includes two main steps: parsing of natural language text and speech generation; natural language text includes steps such as word segmentation, part-of-speech tagging and semantic analysis; speech generation includes spectrum generation, speech synthesis and quality control. Spectrum generation includes generating sound wave waveforms through acoustic models based on text information; speech synthesis includes converting sound wave waveforms into audible sounds, and adjusting parameters such as pitch, speaking speed and intonation of the sounds to generate speech output that meets user needs; these parameter adjustments enable the synthesized speech to adapt to different scenarios and needs, ensuring the naturalness of the speech and clarity of expression; quality control includes quality control in the speech synthesis process, including checks on pronunciation accuracy, natural intonation and moderate speaking speed; quality control measures ensure that the generated speech can clearly convey the content of the explanation and adapt to the needs of various training scenarios.

[0080] Step 6: Integrate the training PPT, the explanation documents and teaching voices corresponding to each training content of the training PPT into training materials. The training PPT includes several pages of PPT, each page of PPT corresponds to an explanation document and teaching voice. The explanation document is a document generated based on the training content recorded in the corresponding page of PPT; the teaching voice is the explanation voice generated based on the corresponding explanation document.

[0081] See also Figure 3 , generating answer information according to the question information, including the following steps:

[0082] Step 1: extracting the question voice in the question information and converting the question voice into question text; the user can ask questions through a microphone or other voice input device;

[0083] Automatic speech recognition (ASR) technology is used to convert the user's voice input into text. Specifically, the captured voice signal is preprocessed, including noise removal and sound quality enhancement, to improve recognition accuracy. In this embodiment, an ASR model is used, such as DeepSpeech or Kaldi, to convert the processed voice signal into text. These models use deep learning technology to model voice and can achieve efficient and accurate voice-to-text conversion.

[0084] Step 2: preprocessing the question text to obtain a clean question text; the preprocessing includes character filtering and noise processing;

[0085] Character filtering in this embodiment is to remove special characters, punctuation marks and extra spaces to ensure that the text contains only meaningful words and phrases; it helps to improve the readability of the text and the accuracy of subsequent processing; noise processing is to use regular expressions or other text processing tools to remove noise in the text; it is convenient to remove any unnecessary information or formatting problems to obtain clean text data.

[0086] The question text is segmented to obtain several independent word units; the independent string is entity recognized to obtain its corresponding entity label; the entity label includes names of people, places, time, etc. When the word unit is recognized as a name of a person, place, or time, the entity label corresponding to the word unit is generated

[0087] In this embodiment, a word segmentation tool (such as NLTK or Jieba) is used to segment the text into independent words and phrases; specifically, Tokens=Tokenizer(Text) is used to extract words and phrases from the text, where Text is the input text, Tokenizer() is the segmentation function in the corresponding word segmentation tool, and Tokens is the words and phrases obtained by segmentation; the named entity recognition (NER, Named Entity Recognition) model is used to identify entity information in the text, such as names, place names, time, etc. Specifically, words or phrases are identified through Entities=NER(Tokens), where NER() is the recognition function corresponding to the NER model, and Entities is the recognition result; the NER model extracts entities with specific meanings by annotating the text to facilitate subsequent analysis and processing.

[0088] Step 4: Vectorize the question text to obtain a query vector; and use the query vector to query the document library to obtain the most similar document vector; the document vector is the vector generated by vectorizing each document in the library, and is pre-generated in the corresponding library; it is convenient for subsequent direct use and improves data processing efficiency.

[0089] Use pre-trained embedding models, such as the BERT model or the GloVe model, to convert the question text into a query vector. Specifically, use Vector=Embedder(Text) to convert the question text into a query vector Vector, where Embedder() is the vector generation function in the corresponding model. Then use retrieval tools, such as FAISS or Annoy, to find the document vector that is most similar to the query vector in the pre-built document library. Specifically, calculate the cosine similarity between the query vector and each document vector; and output the document vector with the highest cosine similarity.

[0090] You can also get several independent word units by segmenting the question text; perform entity recognition on independent character strings to get their corresponding entity tags; use the similarity between independent word units and independent word units of the answer text corresponding to the document vector to verify the accuracy of the document vector; entity tags include names, place names, time, etc. When a word unit is identified as a name, place name, or time, an entity tag corresponding to the word unit will be generated

[0091] In this embodiment, a word segmentation tool such as NLTK word segmentation library or Jieba word segmentation library is used to segment the text into independent words and phrases; specifically, Tokens=Tokenizer(Text) is used to extract words and phrases in the text, where Text is the input text, Tokenizer() is the segmentation function in the corresponding word segmentation tool, and Tokens is the words and phrases obtained by segmentation; the named entity recognition (NER, Named Entity Recognition) model is used to identify entity information in the text, such as names, place names, time, etc. Specifically, words or phrases are identified through Entities=NER(Tokens), where NER() is the recognition function corresponding to the NER model, and Entities is the recognition result; the NER model extracts entities with specific meanings by annotating the text to facilitate subsequent analysis and processing.

[0092] Step 4: Generate answer information based on the query vector and document vector.

[0093] Generate answer information based on query vector and document vector, including:

[0094] Step 1: Input the query vector and the document vector into the answer generation model to obtain the initial answer text; the answer generation model includes the RAG generation model; in this embodiment, the initial answer text is obtained by the pre-trained RAG generation model; the initial answer text is the answer text generated by the model;

[0095] Step 2: Check and revise the grammar of the initial answer text;

[0096] This embodiment detects and corrects grammatical errors by using a language model or rule base; specifically, the answer text is generated by CorrectedText=GrammarCorrection(CorrectedText; GrammarRules); wherein CorrectedText is the generated answer text; GrammarRules is the rule or model for grammar detection; GrammarCorrection() is the corresponding grammar correction function.

[0097] Step three: obtain the semantic association between the initial answer text and the question text; when the semantic association is greater than the set threshold, mark the initial answer text as the answer text; go to step four; otherwise, go to step one; specifically through the pre-trained association model; input the initial answer text and the question text into the association model to obtain the semantic association between the initial answer text and the question text, and the semantic association represents the relevance and consistency between the initial answer text and the question text.

[0098] Step 4: Generate answer voice according to the answer text; integrate the answer text and answer voice into answer information.

[0099] Generating an assessment test paper based on training materials and question information includes the following steps:

[0100] Step 1: Extract question information and knowledge points from each training document in the training materials. Specifically, use the large language model to analyze the training documents and automatically extract key information and core knowledge points. This process includes in-depth understanding of the text content and identification of key concepts; generate several knowledge points K corresponding to the training document D through K=Model (D); where Model () represents the large language model; the extracted knowledge points will be summarized into structured information for subsequent generation of test questions; ensure that the extracted knowledge points cover the main content of the training objectives.

[0101] Step 2: Get the training objectives and generate test questions based on the training objectives and knowledge points; the test questions include single-choice questions, multiple-choice questions and short-answer questions; specifically, generate multiple types of questions based on the training objectives and the extracted knowledge points, including single-choice questions, multiple-choice questions and short-answer questions. Different types of questions can comprehensively evaluate employees' mastery of knowledge points; generate several questions Q corresponding to knowledge point K through Q=Model(K).

[0102] Step 3: Integrate several test questions into an assessment paper.

[0103] Generate a training evaluation based on the assessment test paper, including: obtain the test paper answers of each student, input the test paper answers into the intelligent scoring model to obtain the corresponding test paper score, and use the average score of each test paper score as the training evaluation; the intelligent scoring model is obtained by training the large language model; specifically, use concise and clear answers and complex and ambiguous answers to the same question for training, to ensure that when scoring the corresponding answer information, there will not be a large difference in scores due to similar answers of the students but different answer methods.

[0104] Answer based on the answer information generated by the question information, including:

[0105] Step 1: After the answer information is generated, obtain the question text and extract the answer text and answer voice in the answer information;

[0106] Step 2: Get the current training moment and determine whether the current training moment is a speech node. A speech node is a time node between the corresponding moment when each sentence in the teaching speech ends and the beginning of the next sentence. If yes, proceed to step 3. If not, wait for the arrival of the speech node. When the speech node arrives, proceed to step 3.

[0107] Step 3: Mark the speech node as an interruption node; display the question text and the answer text, and play the answer voice; and obtain the partial explanation document before the speech node, the partial explanation document is the part of the explanation document that has been played corresponding to the current PPT; generate a transition explanation document and a transition teaching voice according to the partial explanation document; the transition explanation document is a summary of the partial explanation document; when the answer voice ends, enter step 4;

[0108] Step 4: Play the transition teaching voice; after the teaching voice is finished playing, jump to the interruption node and continue playing the teaching voice.

[0109] It is understandable that when answering questions in the training module, one should first determine whether the current teaching has been completed. When it has been completed, one should answer; and at the same time summarize the content before the current page of PPT. After the answer is completed, play the summarized content, and then continue to explain the subsequent content; this is conducive to connecting the content before and after answering the question.

[0110] Generate a transitional explanation document and a transitional teaching voice according to a partial explanation document, including: obtaining a partial explanation document and a summary vocabulary ratio range; inputting the partial explanation document and the summary vocabulary ratio range into a large language model to obtain a transitional explanation document, and generate a transitional teaching voice according to the transitional explanation document. The summary vocabulary ratio in this embodiment is a professional vocabulary ratio range of 65%-70%, a common vocabulary ratio of 25%-30%, and an auxiliary vocabulary ratio of 5%-10%; the specific ratio range is set according to experience; it serves to review the previous content and facilitate the connection between the previous and subsequent contents of the answer; it can be understood that the professional vocabulary ratio in the summary vocabulary ratio is higher than the professional vocabulary ratio when the explanation document is generated.

[0111] Generate personalized training materials based on the training evaluation of the test paper, including the following steps:

[0112] Step 1: Obtain the professional vocabulary proportion ZB, general vocabulary proportion PB and auxiliary vocabulary proportion FB of this explanation document; as well as the test score YPi of each student in the training evaluation, where i is the number of each student;

[0113] Step 2: Determine whether the test score YPi of each student is greater than the set training evaluation standard BP; if yes, proceed to step 3; if no, proceed to step 4;

[0114] Step 3: Calculate the adjustment ratio TBi corresponding to the trainee numbered i through the formula TBi = (YPi - BP) / MP × KB1; KB1 is the adjustable ratio 1, and KB1 < min(min(PB, FB), 1 - ZB); Calculate the personalized professional vocabulary ratio GZB through the formula GZB = ZB + TBi; Calculate the personalized common vocabulary ratio GPB through the formula GPB = PB - α1 × TBi; Calculate the personalized auxiliary vocabulary ratio GFB through the formula GFB = FB - α2 × TBi; where α1 and α2 are proportionality coefficients; and α1 + α2 = 1; Jump to Step 5;

[0115] In this embodiment, the specific values of α1 and α2 in Step 3 can be set according to experience, and can be calculated through the formula α1 = PB / (PB + FB), α2 = FB / (PB + FB); or other values, and the specific values are used to adjust the increase and decrease ratios of common vocabulary and auxiliary vocabulary;

[0116] It can be understood that regardless of how the professional vocabulary ratio, common vocabulary ratio, and auxiliary vocabulary ratio increase and decrease, the sum of their corresponding professional vocabulary ratio, common vocabulary ratio, and auxiliary vocabulary ratio is 1. When the professional vocabulary ratio is very small or very large, the corresponding proportionality coefficient of the common vocabulary ratio can be set to a number less than zero, but it is necessary to ensure that α1 + α2 = 1.

[0117] Step 4: Calculate the adjustment ratio TBi corresponding to the trainee numbered i through the formula TBi = (BP - YPi) / MP × KB2; KB2 is the adjustable ratio 2; KB2 < min(1 - max(PB, FB), ZB); Calculate the personalized professional vocabulary ratio GZB through the formula GZB = ZB - TBi; Calculate the personalized common vocabulary ratio GPB through the formula GPB = PB + α1 × TBi; Calculate the personalized auxiliary vocabulary ratio GFB through the formula GFB = FB + α2 × TBi; where α1 and α2 are proportionality coefficients; and α1 + α2 = 1; Enter Step 5;

[0118] Step 5: Generate a personalized explanation document and personalized teaching voice according to the personalized professional vocabulary ratio, personalized common vocabulary ratio, and personalized auxiliary vocabulary ratio, and the corresponding training content; Package the personalized explanation document, personalized teaching voice, and training content into the personalized training materials for the corresponding trainee.

[0119] This embodiment provides personalized training materials for students through the analysis results of the test scores of each student. When a student finds a knowledge point difficult, that is, the learning situation is poor, a simpler explanation text is generated to facilitate the relevant students to carry out supplementary learning after the training; if the student finds the knowledge point simple, a more professional explanation text is generated to facilitate the corresponding students to quickly strengthen the training content. While ensuring the training results, the time of high-quality students is saved, helping them to improve their knowledge level and skills in a targeted manner.

[0120] It is worth noting that this embodiment also includes using the current training evaluation to generate the personalized vocabulary ratio for the next training; specifically, the following steps are included:

[0121] Step 1: Obtain the professional vocabulary proportion ZB, general vocabulary proportion PB and auxiliary vocabulary proportion FB of this explanation document; as well as the test paper score YPi of each student in the training evaluation, where i is the number of each student; determine whether the average score of each student's test paper score is greater than the set training evaluation standard; if yes, proceed to step 2; otherwise, proceed to step 4;

[0122] Step 2: The adjustment ratio TB is calculated by the formula TB=∑((YPi-BP) / MP) / n×KB1; where KB1 is the adjustable ratio 1, and KB1<1-ZB; i is the number of each trainee, and i=1, 2,…, n; n is the total number of trainees; BP is the training evaluation standard; MP is the full score of the training evaluation;

[0123] Step 3: Calculate the professional vocabulary proportion TZB of the next training by the formula TZB=ZB+TB; calculate the general vocabulary proportion TPB of the next training by the formula TPB=PB-α1×TB; calculate the auxiliary vocabulary proportion TFB of the next training by the formula TFB=FB-α2×TB; where α1 and α2 are proportional coefficients; and α1+α2=1; jump to step 6;

[0124] Step 4: Through the formula TB=∑((BP-YPi) / MP) / n×KB2; KB2 is the adjustable ratio 2; KB2<1-max(PB, FB); i is the number of each trainee, and i=1, 2,…, n; n is the total number of trainees; BP is the training evaluation standard; MP is the full score of the training evaluation;

[0125] Step 5: Calculate the professional vocabulary proportion TZB of the next training by the formula TZB=ZB-TB; calculate the general vocabulary proportion TPB of the next training by the formula TPB=PB+α1×TB; calculate the auxiliary vocabulary proportion TFB of the next training by the formula TFB=FB+α2×TB; where α1 and α2 are proportional coefficients; and α1+α2=1; proceed to step 6;

[0126] Step six: Integrate the professional vocabulary ratio, common vocabulary ratio and auxiliary vocabulary ratio of the next training into a personalized vocabulary ratio; use it to generate an explanation document for the next training; it is understandable that since the personnel trained each time in the internal training of the enterprise are not much different, the vocabulary ratio that best suits the corresponding trainees can be obtained through continuous training and adjustment of the vocabulary ratio.

[0127] Part of the data in the above formula is calculated by removing the dimension and taking its numerical value. The formula is a formula closest to the actual situation obtained by software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained through simulation of a large amount of data.

[0128] How this application works:

[0129] The present application generates training materials according to training documents and personalized modification information; conducts training according to the training materials; obtains students' question information during the training process; generates answer information according to the question information; after the answer information is generated, obtains the question text, and extracts the answer text and answer voice in the answer information; if the current training moment is a voice node; displays the question text and the answer text, and plays the answer voice; obtains part of the explanation document before the voice node, and generates a transition explanation document and a transition teaching voice according to the part of the explanation document; when the answer voice ends, plays the transition teaching voice; after the teaching voice is played, jumps to the interruption node and continues to play the teaching voice; generates an assessment paper according to the training materials and question information; generates a training evaluation according to the answers to the paper; generates personalized training materials according to the training evaluation of the assessment paper; realizes the whole process of intelligent training; by promptly answering students' question information, ensures the students' learning effect in this training and the continuity of the training; thereby increases the training effect of this training.

[0130] The above embodiments are only used to illustrate the technical method of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical method of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present application.

Claims

1. An intelligent training system based on an end-to-end multimodal teaching model, include: Training material generation module, question and answer module, evaluation module, training module and interaction module; characterized in that, The interactive module is used to extract training documents, personalized modification information, question information and test answers from the training materials; The training material generation module generates training materials according to the training documents and the personalized modification information; the training material generation module generates training materials according to the training documents and the personalized modification information, including the following steps: Step 1: extract training documents from the training materials, pre-process the training documents to obtain key content, and input the key content into the large language model to obtain a PPT outline; the training materials include training documents, explanation documents and teaching voices; the training documents include a PPT outline and training content; Step 2: Obtain the user's personalized modification information for the PPT outline; determine whether the personalized modification information is "none"; if yes, mark the PPT outline as a confirmed PPT outline; proceed to step 3; if no, modify the PPT outline according to the personalized modification information, and proceed to step 3; Step 3: Input the confirmed PPT outline into the large language model to obtain training content, wherein the training content includes training text and training pictures; generate a training PPT according to the PPT outline, training text and training pictures; Step 4: Obtain the training text and training pictures of each page in the training content; input them into the large language model to obtain the corresponding explanation document; Step 5: Use Text to Speech technology to generate the explanation document into teaching voice; Step 6: Integrate the training PPT, as well as the explanation documents and teaching voices of each training content in the training PPT into training materials; The question-answering module generates answer information according to the question information; The evaluation module: generates an evaluation test paper based on the training materials and question information; generates a training evaluation based on the evaluation test paper; The training module: performs training according to the training materials; provides answers according to the answer information generated by the question information; and generates personalized training materials according to the training evaluation of the evaluation test paper, and sends the personalized training materials to the corresponding user; the generation of personalized training materials according to the training evaluation of the evaluation test paper includes the following steps: Step 1: Obtain the professional vocabulary proportion ZB, general vocabulary proportion PB and auxiliary vocabulary proportion FB of this explanation document; as well as the test score YPi of each student in the training evaluation, where i is the number of each student; Step 2: Determine whether the test score YPi of each student is greater than the set training evaluation standard BP; if yes, proceed to step 3; if no, proceed to step 4; Step 3: Calculate the adjustment ratio TBi corresponding to the student numbered i by the formula TBi=(YPi-BP) / MP×KB1; calculate the personalized professional vocabulary ratio GZB by the formula GZB=ZB+TBi; calculate the personalized general vocabulary ratio GPB by the formula GPB=PB-α1×TBi; calculate the personalized auxiliary vocabulary ratio GFB by the formula GFB=FB-α2×TBi; where α1 and α2 are proportional coefficients; and α1+α2=1; jump to step 5; Step 4: Calculate the adjustment ratio TBi corresponding to the student numbered i by the formula TBi=(BP-YPi) / MP×KB2; calculate the personalized professional vocabulary ratio GZB by the formula GZB=ZB-TBi; calculate the personalized general vocabulary ratio GPB by the formula GPB=PB+α1×TBi; calculate the personalized auxiliary vocabulary ratio GFB by the formula GFB=FB+α2×TBi; where α1 and α2 are proportional coefficients; and α1+α2=1; proceed to step 5; Step 5: Generate personalized explanation documents and personalized teaching voices according to the proportion of personalized professional vocabulary, personalized general vocabulary, personalized auxiliary vocabulary, and corresponding training content; package the personalized explanation documents, personalized teaching voices, and training content into personalized training materials, and send them to corresponding students.

2. According to claim 1, an intelligent training system based on an end-to-end multimodal teaching model, It is characterized in that The step of generating answer information according to the question information comprises the following steps: Step 1: extracting the question voice in the question information, and converting the question voice into question text; Step 2: preprocessing the question text to obtain a clean question text; the preprocessing includes character filtering and noise processing; Step 3: Vectorize the question text to obtain the query vector; then use the query vector to search the document library to obtain the most similar document vector; Step 4: Generate answer information based on the query vector and document vector.

3. According to claim 2, an intelligent training system based on an end-to-end multimodal teaching model, It is characterized in that The generating of answer information according to the query vector and the document vector includes: Step 1: Input the query vector and the document vector into the answer generation model to obtain the initial answer text; the answer generation model includes the RAG generation model; Step 2: Check and revise the grammar of the initial answer text; Step 3: Obtain the semantic relevance between the initial answer text and the question text; when the semantic relevance is greater than the set threshold, mark the initial answer text as the answer text; proceed to step 4; otherwise, proceed to step 1; Step 4: Generate answer voice according to the answer text; integrate the answer text and answer voice into answer information.

4. According to claim 3, an intelligent training system based on an end-to-end multimodal teaching model, It is characterized in that The step of answering the question based on the answer information generated includes: Step 1: After the answer information is generated, obtain the question text and extract the answer text and answer voice in the answer information; Step 2: Get the current training moment, and determine whether the current training moment is a voice node. If yes, proceed to step 3; if no, wait for the arrival of the voice node; when the voice node arrives, proceed to step 3; Step 3: Mark the speech node as an interruption node; display the question text and the answer text, and play the answer voice; and obtain the partial explanation document before the speech node, and generate a transition explanation document and a transition teaching voice according to the partial explanation document; when the answer voice ends, proceed to step 4; Step 4: Play the transition teaching voice; after the teaching voice is finished playing, jump to the interruption node and continue playing the teaching voice.

5. According to claim 4, an intelligent training system based on an end-to-end multimodal teaching model, It is characterized in that The step of generating a transitional explanation document and a transitional teaching voice according to a partial explanation document includes: Obtain partial explanation documents and summary vocabulary ratio ranges; input partial explanation documents and summary vocabulary ratio ranges into the large language model to obtain transition explanation documents, and generate transition teaching voices according to the transition explanation documents.

6. The intelligent training system based on the end-to-end multimodal teaching model according to claim 1, It is characterized in that The step of generating the assessment test paper according to the training materials and question information includes the following steps: Step 1: Extract question information and knowledge points from each training document in the training materials Step 2: Obtain training objectives, and generate test questions based on the training objectives and knowledge points; the test questions include single-choice questions, multiple-choice questions, and short-answer questions; Step 3: Integrate several test questions into an assessment paper.

7. The intelligent training system based on the end-to-end multimodal teaching model according to claim 1, It is characterized in that Generating a training evaluation based on the test paper includes: The test answers of each student are obtained, and the test answers are input into the intelligent scoring model to obtain the corresponding test scores, and the training evaluation is integrated according to the test scores.

Images