An intelligent answering method based on multi-source heterogeneous knowledge in training and testing

Abstract

The present invention relates to an intelligent answering method based on multi-source heterogeneous knowledge of training and examination. The method is based on multi-source heterogeneous data such as training and examination teaching materials, training and examination images, training and examination videos, training and examination documents, training and examination voices, and training and examination structured data. The training and examination text knowledge is extracted using a training and examination extraction model. By constructing a training and examination multimodal model, a training and examination association model, a training and examination voice-text conversion model, and a training and examination 3D instructor model, it supports the input of multi-source heterogeneous questions such as images, documents, texts, and voices, and can achieve accurate answering of professional training and examinations such as training and examination analysis, training suggestions, and training plans for individuals and groups, support giving analysis results through images and tables, and provide professional explanations of training and examination 3D digital instructors. Voice and character image cloning can be performed according to the preferences of students to improve the training interest of students, and it has a high market promotion value in the field of intelligent training. The present invention can achieve accurate answering of professional training and examination knowledge.

Description

Technical Field

[0001] The present invention relates to the field of intelligent training technology, and in particular to an intelligent answering method based on training and testing multi-source heterogeneous knowledge. Background Art

[0002] Due to the different domestic and international scenarios, the composition of training personnel, training equipment, and examination training subjects is different. There are obvious differences in the research points and technologies in physical training, and there are also differences in physical answering business.

[0003] At present, the dialogue in the training field is mainly based on routine questions, facing the pain points of low professionalism and accuracy of intelligent dialogue, uncontrollable output, high cost of training models, and low economic efficiency. It has high requirements for hardware environment deployment, cannot provide accurate answers to physical training and tests, lacks professional reasoning capabilities for physical training and tests, and cannot realize professional explanations by 3D digital instructors for training and tests, and cannot meet the requirements of professionalism, flexibility, and personalization. Summary of the invention

[0004] The purpose of the present invention is to provide an intelligent answering method based on multi-source heterogeneous knowledge of training and testing, so as to achieve professional and accurate answering of training and testing knowledge in the professional field of physical training.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] An intelligent answering method based on multi-source heterogeneous knowledge of training and testing, comprising:

[0007] Acquire multi-source heterogeneous knowledge of training and testing, extract the multi-source heterogeneous knowledge of training and testing, acquire training and testing text data and mark questions and answers;

[0008] Constructing a training-testing multimodal model, inputting the training-testing question into the training-testing multimodal model, and obtaining a first answer and a relational knowledge graph, wherein the training-testing multimodal model is trained and tuned using the training-testing text data;

[0009] Constructing a training-examination association mining model, and inputting the training-examination question into the training-examination association mining model to obtain a second answer;

[0010] The first answer and the second answer are combined, converted into Embeddings, and summed to output a unified answer to the question;

[0011] For training texts with fixed formats such as training suggestions, training analysis reports, and training prescriptions, the Naive Bayes algorithm is used to train to obtain image and table analysis result templates. The answer values ​​and template contents are matched by establishing a numerical and entity GNN graph neural network to optimize the entities and values ​​in the image and table analysis result templates to output correct image and table analysis results.

[0012] According to the answers to the questions, a training and test answer video is generated through the 3D digital instructor model to interact with the virtual 3D digital instructor.

[0013] Optionally, extracting the multi-source heterogeneous knowledge for training and testing includes:

[0014] Acquire multi-source heterogeneous knowledge of training and testing, wherein the multi-source heterogeneous knowledge of training and testing includes training and testing teaching materials, training and testing images, training and testing videos, training and testing documents, training and testing voices, and training and testing structured data;

[0015] Extracting the multi-source heterogeneous knowledge of training and testing through a training and testing extraction model to obtain the training and testing text data;

[0016] The text-based large language question-answering model is used as the first task multimodal model of training and testing to extract data from the training and testing structured data to generate training and testing text data Y1;

[0017] Use the image recognition algorithm as the second task model of the training test to perform content recognition on the training test image and output the training test text data Y2;

[0018] Use the video recognition algorithm as the third task model of training and testing to perform content recognition on the training and testing video, and generate and output training and testing text data Y3;

[0019] Using the text recognition algorithm as the fourth task model of the training and test to extract data from the training and test teaching materials, and generate training and test text data Y4;

[0020] The document large language question answering model is used as the fifth training and test task multimodal model to extract the document content from the training and test document to generate the fifth training and test text data Y5;

[0021] The training and test speech-to-text conversion model is used as the sixth task model of the training and test to extract data from the training and test speech and generate training and test text data Y6.

[0022] Optionally, constructing the training-testing multimodal model includes:

[0023] Based on the questions and answers in the training and test text data, the training and test first task multimodal model and the training and test fifth task multimodal model are trained by Markov decision process and reinforcement learning to obtain a training and test model MM;

[0024] The training-testing model MM is integrated with the second task model, the third task model and the fourth task model to obtain the training-testing multimodal model.

[0025] Optionally, the objective function in the process of training the first task multimodal model and the fifth task multimodal model through the Markov decision process is:

[0026]

[0027] Among them, X max represents the number of the first task multimodal model M1 and the fifth task multimodal model M5, Y max represents the number of test questions, I uv Indicates whether the test question v is wrong when using the u model. represents the number of correct answers to questions v by model u, represents the response time of model u to answer question v, W 0 , W 1 They represent the weight values ​​assigned according to the correct number and response time respectively, and T represents the total time of the correct number of questions and answers.

[0028] Optionally, inputting the training question into the training-test association mining model to obtain a second answer includes:

[0029] Based on the training and test text data, obtaining a relational knowledge graph;

[0030] Calculate all path sets in the relational knowledge graph, and use GCN to remove erroneous relations and attributes in the relational knowledge graph;

[0031] The Encoder-Decoder framework is combined with GNN to reconstruct the entity relationships in the relational knowledge graph and optimize the relational knowledge graph;

[0032] Calculate the optimized relational knowledge graph according to the similarity to obtain the optimal similar vertex;

[0033] According to the best similar vertex, obtaining a path from the best similar vertex to a leaf edge node;

[0034] The similarity between the result word vector of the path and the question vector is calculated, and the minimum similarity value is the second answer.

[0035] Optionally, obtaining the relational knowledge graph includes:

[0036] The training text data is divided into P 1 , P 2 , P 3 ...P i ...P n , n is the number of punctuation marks, P i By D i1 ,D i2 ,……D ij ...D im Composition, m is P i Number of words;

[0037] Use the TextCNN algorithm to obtain t classification noun training word embedding vectors;

[0038] Will Input the BiLSTM model to obtain the probability value of the word embedding vector classification noun;

[0039] Pearson correlation coefficient was used to calculate D ij With P i -D ij , D ij The correlation with t classification nouns, according to the correlation, filter out the probability values ​​that do not meet the correlation, input the result into the CRF model to obtain the label sequence with the highest probability value, and obtain the training and test entities, training and test attributes and training and test relationships;

[0040] The relational knowledge graph is obtained by combining the training and testing entities, the training and testing attributes and the training and testing relationships.

[0041] Optional, generate training and test answer videos and interact with virtual 3D digital instructors, including:

[0042] Obtaining a plurality of groups of picture prompt texts and a plurality of groups of description voices corresponding to the pictures for the answers to the questions through the training-test multimodal model and the training-test voice-to-text conversion model;

[0043] Using a random forest algorithm to obtain picture prompt texts and picture description voices of the same category as the answer to the question and perform similarity calculation to obtain a group of picture prompt texts and a group of picture description voices;

[0044] Processing the set of picture prompt texts based on the generator network and the discriminator network and combining them with the set of picture description voices to obtain the training and test answer video;

[0045] MHFormer is used to generate 3D human skeleton points, and the key points are fitted using the human body model. The optimal model is determined by the naive Bayes algorithm to establish a digital 3D instructor model.

[0046] The answer to the question is converted into a voice with a preset timbre by using the training and test voice-to-text conversion model, and voice features are extracted;

[0047] The speech feature, the posture model and the digital 3D instructor model are integrated and input into a multi-layer perceptron to obtain the virtual 3D digital instructor;

[0048] Combine the training and test answer video with the virtual 3D digital instructor for interactive dialogue.

[0049] Optionally, obtaining the training and test answer video includes:

[0050] The set of image prompt texts are respectively input into a generator network for generating an image foreground and a generator network for generating an image background, and a Self-Attention mechanism is introduced to generate foreground and background text images;

[0051] Using a discriminator network to discriminate the generated foreground and background text images and the real foreground images and background images, and generate the set of images corresponding to the image prompt text;

[0052] Calculate similarity between the pictures corresponding to the set of picture prompt texts, and obtain target training and test videos from multi-source heterogeneous knowledge of training and test;

[0053] The target training test video and the set of picture description voices are synthesized to obtain the training test answer video.

[0054] Optionally, the training process of using a discriminator network to discriminate the generated foreground and background text images from real foreground images and background images is:

[0055]

[0056] Among them, x represents the training test picture used for training, e represents the training test answer text, Q 1 represents the training foreground discriminator network, Q 1 (x) represents the discriminator output of the training and test foreground real images, Q 2 represents the training background discriminator network, Q 2 (x) represents the discriminator output of the real background image of the training test, K 1 represents the training and testing prospect generator network, K 1 (e) represents the training and testing foreground images generated by the model, K 2 represents the training background generator network, K 2 (e) represents the training background image generated by the model, E x～p(x) represents the expected distribution of training and test image sample data, E e～p(e) Represents the expectation of the distribution of sample data of training and test answers, through the joint Maximize the objective function, joint Minimize the objective function and get the optimal parameters.

[0057] The beneficial effects of the present invention are:

[0058] The present invention has high commercial and practical value, realizes an intelligent answering method based on multi-source heterogeneous knowledge of training and testing, and the method can realize professional and accurate answering of training and testing knowledge by establishing a training and testing model. The training and testing model includes a training and testing extraction model, a training and testing multimodal model, a training and testing association mining model, a training and testing speech-to-text conversion model, and a training and testing 3D digital instructor generation model, which expands the generalization ability of the model in answering multi-source heterogeneous knowledge of training and testing, improves the answering reasoning ability of the model, can solve the pain points of low professionalism and accuracy of intelligent answering, and realizes controllable output and low cost of training models. The method has low requirements for hardware environment deployment and is more economical and efficient, has strong professional reasoning ability for training and testing, and can also realize professional explanation of 3D digital instructors for training and testing, meets the requirements of professionalism, flexibility, personalization, etc., can provide users with efficient and intelligent dialogue experience, help them quickly master training and testing knowledge, improve training and testing capabilities, provide managers with efficient and intelligent management means, and has high market promotion value in the field of intelligent training. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0060] Figure 1 A business process diagram of an intelligent answering method based on training and testing multi-source heterogeneous knowledge in an embodiment of the present invention;

[0061] Figure 2 The present invention is a schematic diagram of an intelligent answering method based on training and testing multi-source heterogeneous knowledge according to an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0064] like Figure 1 and Figure 2As shown, an intelligent answering method based on multi-source heterogeneous knowledge of training and testing can achieve accurate answering of training and testing knowledge in the field of physical training by establishing a training and testing model. The training and testing model includes a training and testing extraction model, a training and testing multimodal model, a training and testing association mining model, a training and testing speech-to-text conversion model, and a training and testing 3D digital instructor model, which includes the following steps:

[0065] Step 1: The sources of multi-source heterogeneous knowledge of training and examination include multi-source heterogeneous data such as training and examination teaching materials, training and examination images, training and examination videos, training and examination documents, training and examination voices, and training and examination structured data. The training and examination extraction model is used to extract the multi-source heterogeneous knowledge of training and examination into training and examination text data and mark the questions and answers.

[0066] The specific steps for constructing a training and test extraction model to extract multi-source heterogeneous training and test knowledge into training and test text data are as follows:

[0067] Step 1: Select a mature text-based large language question-answering model as the first multimodal model M1 for the training and examination task, select a mature image recognition algorithm as the second image model M2 for the training and examination task, select a mature video recognition algorithm as the third video model M3 for the training and examination task, select a mature text recognition algorithm as the fourth text recognition model M4 for the training and examination task, and select a mature document-based large language question-answering model as the fifth multimodal model M5 for the training and examination task. M1 supports long text question-answering function, M2 supports content recognition of training and examination images, M3 supports action recognition of training and examination videos, M4 supports text recognition of professional teaching materials for sports training and examination, and M5 supports document question-answering function;

[0068] Step 2: Extract data from the training and test structured data using the training and test first task multimodal model M1 to generate training and test text data Y1;

[0069] Step 3: Use the training and examination second task image model M2 to perform content recognition on the training and examination image, such as outputting the training and examination text data Y2 such as the trainee's name, training and examination action, training and examination equipment, training and examination subject, training and examination result, training and examination suggestion, etc.;

[0070] Step 3: Use the training and examination third task video model M3 to perform content recognition on the training and examination video, such as outputting the training and examination text data Y3 such as the trainee's name, training and examination actions, training and examination equipment, training and examination subjects, training and examination results, and training and examination suggestions;

[0071] Step 4: Perform text recognition on the professional teaching materials of sports training and examination through the fourth task text recognition model M4 of training and examination to generate professional text data Y4 of training and examination;

[0072] Step 4: Use the fifth task multimodal model M5 of the training and examination to extract the content of the training and examination documents in word, pdf and other formats to generate training and examination text data Y5;

[0073] Step 5: Generate training text data Y6 for the training speech using the training speech-to-text conversion model;

[0074] Step 7: Select test samples from the training and examination professional text data Y1, Y2, Y3, Y4, Y5, and Y6 and mark them as questions and answers recorded as Q1.

[0075] Step 2: Build a training-testing multimodal model and use the training-testing text data obtained in step 1 for training and tuning;

[0076] The specific steps to build a training-testing multimodal model are as follows:

[0077] Step 1: Use commas, periods, and semicolons to separate the training text data obtained in step 1 into P 1 , P 2 , P 3 ...P i ...P n , n is the number of the above punctuation marks, P i By (D i1 ,D i2 ,……D ij ...D im ),m is P i The number of words is calculated by using the TextCNN algorithm to train word embedding vectors according to t categorical nouns such as "name", "training test subject", "training test equipment", "training test score", and "training suggestion". Input the BiLSTM model to obtain the probability value of the word embedding vector classification noun, and use the Pearson correlation coefficient to calculate D ij With P i -D ij , D ij The correlation with t classification nouns is filtered out according to the correlation results to obtain the probability value, and then the result is input into the CRF model to obtain the label sequence with the highest probability value, so as to obtain the training entity and training attribute. The above classification nouns are replaced with nouns such as "belong to", "subclass", "obtain", and "give". Repeating the above process can obtain the training relationship data, which can be established <Z 1 ,Z 2 ,Z 3 >t*(t-1)*(t-2) training-test relationships, Z 1 ,Z 2 ,Z 3 For training and testing entities, Z 1 ,Z 2 ,Z 3 ∈{student name, training subject, training equipment, training results, training suggestions, ...} and Z 1 ,≠Z 2 ≠Z 3, the training-examination relationship combines the training-examination entities and attributes to construct a relational knowledge graph M0 and perform 3D visualization;

[0078] Step 2: According to the types of training and test questions, such as pictures, videos, word, and pdf documents, select the corresponding models from the second task image model M2, the third task video model M3, and the fourth task text model M4 to automatically perform parsing and store the content in the answer library. The text questions are respectively input into the first task multimodal model M1 and the fifth task multimodal model M5 of the training and test. The obtained content results are integrated with the answer library results to obtain the training and test answers. The question and answer Q1 obtained in step 7 of step 1 are used to train the first task multimodal model M1 and the fifth task multimodal model M5 of the training and test. The parameter adjustment process of M1 and M5 can be regarded as a multi-agent Markov decision process. Multi-agent reinforcement learning is used to realize competition between models, promote balanced integration, and adjust parameters to obtain the training and test model MM.

[0079] The objective function is defined as:

[0080] Among them, X max represents the number of training and testing the first task multimodal model M1 and the fifth task multimodal model M5, Y max represents the number of test questions, I uv Indicates whether the test question v is wrong when using the u model. The wrong value is 0 and the correct value is 1. represents the number of correct answers to questions v by model u, represents the response time of model u to answer question v, W 0 , W 1 Respectively represent the distribution weight values ​​according to the correct number and response time, T represents the total time for the correct number of questions and answers;

[0081] The maximum cumulative expected return of all agents is F;

[0082] The loss function is: = (-1) × the total time to answer the correct questions, Q ou (s,a,l) is the expected return value, l represents the network parameter, a represents the action of the model in the question-answer state s, and c represents the batch size. Margin is a margin parameter, the default value is 0;

[0083] Step 3: Training and testing question Q1 input model MM by scoring the output answer, and introduce reinforcement learning to fine-tune the training and testing model MM. The reward function is: 1 =R 1 -a1 R 1 KL -b 1 R 1 KL -c 1 R 1 KL , R 1 is the reward function of the multimodal model MM, R 1 KL is the penalty term of KL divergence, a 1 is the random fine-tuning parameter, b 1 is the fine-grained fine-tuning parameter, c 1 In order to fine-tune the parameters in a coarse-grained manner, the training-test model MM integrates M2, M3, and M4 to select the corresponding model according to the input type. The training-test multimodal model M is obtained by convergence and fusion. The training-test questions are input into the training-test multimodal model M to obtain the first answer.

[0084] Step 3: Use the training-examination association mining model to expand the model's question-answering reasoning ability, so as to achieve question-answering of training-examination analysis, training suggestions, training plans, etc. for individuals and groups, and display the analysis results in images and tables;

[0085] The specific steps to expand the model's question-answering reasoning capabilities using the training-test association mining model are as follows:

[0086] Step 1: Calculate the set of all paths in the knowledge graph M0, use GCN to detect errors in entity relationships in M0, and remove incorrect relationships and attributes;

[0087] Step 2: Use the Encoder-Decoder framework and introduce GNN to reconstruct the entity relationships in M0 and continuously optimize the relational knowledge graph M0;

[0088] The new node calculation is expressed as:

[0089] GNN(A,H,W)=ReLU(AHW)=max(0,AHW)

[0090] Among them, A represents the adjacency matrix, H represents the current node, and W represents the learned parameters;

[0091] Step 3: Calculate the rela of all graph vertices in the training and examination knowledge graph M0. According to the rela value, the correlation between training and examination scores, training and examination plans, training and examination subjects, etc. can be determined, and the model reasoning ability can be expanded to recommend the best association relationship and obtain the best answer path, so as to realize the question and answer of personal training and examination analysis, training suggestions, training plans, etc.

[0092]

[0093] Among them, V i 、V j is the vertex entity vector of the training knowledge graph M0, n is the number of graph vertices, V i The mean value, V j for The mean of

[0094] Step 4: Calculate the entity word vector through word2vec and calculate the vector similarity Vecsim of the question and training knowledge graph vertices to obtain the optimal similar vertex.

[0095]

[0096] Among them, vec i and vec j are two word vectors, and m is the dimension of the word vector;

[0097] According to the best similar vertex, find the path from the vertex to all the leaf edge nodes of the graph. The number of paths is recorded as ys. For all entities, attributes, and relationships on each path, the result word vector is formed, and the similarity Vecsim1 between them and the question vector is calculated respectively. The minimum value is the best answer, and the best answer is the second answer.

[0098]

[0099] Among them, vec i is the problem vector, vec j is vs result word vector, m is the dimension of word vector;

[0100] Step 5: Convert the answer obtained in step 4 of step 3 and the answer obtained by the training-test multimodal model M in step 2 into Embedding and sum them up to output a unified answer to the question;

[0101] Step 6: For training text contents with fixed formats such as training suggestions, training analysis reports, and training prescriptions, the naive Bayes algorithm is used to train to obtain image and table analysis result templates. The answer values ​​in step 5 and the template content are used to establish a value and entity GNN graph neural network for value and entity matching, to optimize the entities and values ​​in the image and table analysis result templates to output the correct image and table analysis results.

[0102] Step 4: Build a training and test voice-to-text conversion model to achieve flexible conversion between training and test voice and text. It can convert training and test question voice into question text content, and convert answer text into answer voice, and support voice cloning;

[0103] Step 1: Extract MFCC features from speech and use neural network to train to get text;

[0104] Step 2: Students can clone and output their favorite voices. By using WaveNet to realize text-to-speech CC, CC voice is modeled by autoencoders (VAEs) on mel spectrograms to obtain CC1. CC voice is obtained by Fourier transform. CC1 and CC2 are expanded based on SR data to extract clean content information, and initial weights are assigned to them. They are input into the Encoder, and IPA alignment is performed with a custom voice selected by the student to fit the timbre. Then, the output is output through the Decoder. The discriminator in the generative adversarial network (GAN) is used to judge the output, and the initial weights of CC1 and CC2 are trained and adjusted to obtain the optimal parameters, and finally a high-precision timbre similar to the custom voice is generated.

[0105] Step 3: Use the training text and training voice marked in step 1 to input the above steps 1 and 2 for training and tuning;

[0106] The loss function is:

[0107]

[0108] Among them, L mel represents the loss of the mel spectrogram, L dur Indicates the loss of duration, L pitch represents the loss of pitch predictor, L KL represents the loss of KL divergence, L fm represents the loss of feature matching, is a hyperparameter;

[0109] Step 4: Calculate the Jaccard similarity of the custom voice and the cloned voice in the training voice emotion and intonation, use the LSTM network to train the custom voice emotion and intonation separately to obtain the voice emotion and intonation to be synthesized, and determine the weight distribution ratio between the voice emotion and intonation to be synthesized and the cloned voice obtained in step 2 according to the similarity value, and then synthesize them to make the voice more realistic.

[0110] Step 5: Support the input of text, documents, pictures, videos and other content. Generate the training and examination answer video and virtual 3D digital instructor through the training and examination 3D digital instructor model according to the generated answer text content. The 3D digital instructor can interact with the answer according to the voice and can give voice explanations according to the training and examination answer video;

[0111] The specific steps for generating the training and test answer video and the virtual 3D digital instructor through the training and test 3D digital instructor model according to the generated answer text content are as follows:

[0112] Step 1: The answer text output in step 3, step 5, can be trained by step 2 for the multimodal model and step 4 for the speech-to-text conversion model to obtain nn groups of picture prompt texts and nn groups of pictures corresponding to nn description voices, where nn represents the number of picture groups. The random forest algorithm is used to obtain picture prompt texts and picture description voices that belong to the same category as the answer text. Then, the similarity Vecsim1 in step 3, step 4 is used to calculate a group of picture prompt texts and picture description voices. A group of picture prompt texts includes mm picture prompt texts, where mm represents the number of picture prompt texts. The mm image prompt texts are respectively input into the generator network for generating the picture foreground. A generator network for generating picture backgrounds is used, and a Self-Attention mechanism is introduced to generate foreground and background text pictures. The generated foreground and background pictures are distinguished from the real foreground and background pictures by using a discriminator network. The two are confronted with each other to generate text foreground pictures and text background pictures corresponding to the text. The foreground and background pictures are combined to finally obtain mm pictures corresponding to the text. The similarity Vecsim1 in step 4 of step 3 is used to calculate the mm pictures. The target training video is obtained from the training and test multi-source heterogeneous knowledge. The mm pictures, the target training video and the picture description voice are synthesized to obtain the training and test answer video.

[0113] The training process can be expressed as:

[0114]

[0115] Among them, x represents the training test picture used for training, e represents the training test answer text, Q 1 represents the training foreground discriminator network, Q 1 (x) represents the discriminator output of the training and test foreground real images, Q 2 represents the training background discriminator network, Q 2 (x) represents the discriminator output of the real background image of the training test, K 1 represents the training and testing prospect generator network, K 1 (e) represents the training and testing foreground images generated by the model, K 2 represents the training background generator network, K 2 (e) represents the training background image generated by the model, E x～p(x) represents the expected distribution of training and test image sample data, E e～p(e) Represents the expectation of the distribution of sample data of training and test answers, through the joint Maximize the objective function, joint Minimize the objective function and obtain the optimal parameters;

[0116] Step 2: Select a person picture and use MHFormer to generate 3D human skeleton points, use the human body model to fit the key points, use the naive Bayes algorithm to determine the optimal model, and build a digital 3D instructor model;

[0117] The 3D digital instructor constructs Q(φ,μ,υ) and can be defined as:

[0118] Q(φ,μ,υ)=LBS(F(φ,μ,υ),V(φ),μ,W)+τ

[0119] Among them, φ represents body shape parameters, μ represents posture parameters, υ represents expression parameters, τ represents leg parameters, W represents deformation weight, V(φ) represents key point coordinates obtained from the model shape, LBS can realize matrix transformation, and Q(φ,μ,υ) outputs the three-dimensional coordinates of each vertex;

[0120]

[0121] in, represents the average template shape, H k Represents the body shape correction data, H l Represents attitude correction data, H j Represents expression correction data, Indicates expression change parameters;

[0122] Step 3: The text answer is converted into a timbre similar to the student's customized voice through the training and test voice-to-text conversion model in step 4. The voice features are extracted from the timbre and the posture model is trained. The posture model can learn the consistency between the eyes, gestures, and mouth movements and the voice content through a neural network. The voice features and posture model are combined with the digital 3D instructor model obtained in step 2 and input into the trained multi-layer perceptron. Through the neural radiation field technology, the voice is connected to the digital instructor to obtain a virtual digital training instructor with realistic lip shape and movements.

[0123] Step 4: The virtual digital training instructor explains the answers and presents them through the training and test answer video in step 1.

[0124] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. An intelligent answering method based on training and testing multi-source heterogeneous knowledge, characterized in that: include: Acquire multi-source heterogeneous knowledge of training and testing, extract the multi-source heterogeneous knowledge of training and testing, acquire training and testing text data and mark questions and answers; Constructing a training-testing multimodal model, inputting the training-testing questions into the training-testing multimodal model, selecting a corresponding model according to different input types, and obtaining a first answer, wherein the training-testing multimodal model is trained and tuned using the training-testing text data; Constructing a training-examination association mining model, and inputting the training-examination question into the training-examination association mining model to obtain a second answer; Combine the first answer and the second answer to obtain an answer to the question; Generate a training and test answer video through the 3D digital instructor model according to the answer to the question and interact with the virtual 3D digital instructor; Extracting the multi-source heterogeneous knowledge of the training and testing includes: Acquire multi-source heterogeneous knowledge of training and testing, wherein the multi-source heterogeneous knowledge of training and testing includes training and testing teaching materials, training and testing images, training and testing videos, training and testing documents, training and testing voices, and training and testing structured data; Extracting the multi-source heterogeneous knowledge of training and testing through a training and testing extraction model to obtain the training and testing text data; The text-based large language question-answering model is used as the first task multimodal model of training and testing to extract data from the training and testing structured data to generate training and testing text data Y1; Use the image recognition algorithm as the second task model of the training test to perform content recognition on the training test image and output the training test text data Y2; Use the video recognition algorithm as the third task model of training and testing to perform content recognition on the training and testing video, and generate and output training and testing text data Y3; Using the text recognition algorithm as the fourth task model of the training and test to extract data from the training and test teaching materials, and generate training and test text data Y4; The document large language question answering model is used as the fifth training and test task multimodal model to extract the document content from the training and test document to generate the fifth training and test text data Y5; The training test speech-to-text conversion model is used as the sixth task model of the training test to extract data from the training test speech to generate training test text data Y6; Constructing the training-testing multimodal model includes: Based on the questions and answers in the training and test text data, the training and test first task multimodal model and the training and test fifth task multimodal model are trained by Markov decision process and reinforcement learning to obtain a training and test model MM; The training-testing model MM is integrated with the second task model, the third task model and the fourth task model to obtain the training-testing multimodal model; Inputting the training question into the training-test association mining model to obtain a second answer includes: Based on the training and test text data, obtaining a relational knowledge graph; Calculate all path sets in the relational knowledge graph, and use GCN to remove erroneous relations and attributes in the relational knowledge graph; The Encoder-Decoder framework is combined with GNN to reconstruct the entity relationships in the relational knowledge graph and optimize the relational knowledge graph; Calculate the optimized relational knowledge graph according to the similarity to obtain the optimal similar vertex; According to the best similar vertex, obtaining a path from the best similar vertex to a leaf edge node; Calculate the similarity between the result word vector of the path and the question vector, and the minimum similarity value is the second answer; Generate training and test answer videos and interact with virtual 3D digital instructors, including: Obtaining a plurality of groups of picture prompt texts and a plurality of groups of description voices corresponding to the pictures for the answers to the questions through the training-test multimodal model and the training-test voice-to-text conversion model; Using a random forest algorithm to obtain picture prompt texts and picture description voices of the same category as the answer to the question and perform similarity calculation to obtain a group of picture prompt texts and a group of picture description voices; Processing the set of picture prompt texts based on the generator network and the discriminator network and combining them with the set of picture description voices to obtain the training and test answer video; MHFormer is used to generate 3D human skeleton points, and the key points are fitted using the human body model. The optimal model is determined by the naive Bayes algorithm to establish a digital 3D instructor model. The answer to the question is converted into a voice with a preset timbre by using the training and test voice-to-text conversion model, and voice features are extracted; The speech features, the posture model and the digital 3D instructor model are integrated and input into a multi-layer perceptron to obtain the virtual 3D digital instructor; Combine the training and test answer video with the virtual 3D digital instructor for interactive dialogue.

2. The intelligent answering method based on training and testing multi-source heterogeneous knowledge according to claim 1 is characterized in that: The objective function in the process of training the first task multimodal model and the fifth task multimodal model through the Markov decision process is: Among them, X max represents the number of the first task multimodal model M1 and the fifth task multimodal model M5, Y max represents the number of test questions, I uv Indicates whether the test question v is wrong when using the u model. represents the number of correct answers to questions v by model u, It represents the response time of model u to answer question v, W0 and W1 represent the weight values ​​assigned according to the correct number and response time respectively, and T represents the total time of the correct number of questions and answers.

3. The intelligent answering method based on training and testing multi-source heterogeneous knowledge according to claim 1 is characterized in that: Acquiring the relational knowledge graph includes: The training text data is divided into P1, P2, P3...P i ...P n , n is the number of punctuation marks, P i By D i1 ,D i2 ,……D ij ...D im Composition, m is P i Number of words; Use the TextCNN algorithm to obtain t classification noun training word embedding vectors; Will Input the BiLSTM model to obtain the probability value of the word embedding vector classification noun; Pearson correlation coefficient was used to calculate D ij With P i -D ij , D ij The correlation with t classification nouns, according to the correlation, filter out the probability values ​​that do not meet the correlation, input the result into the CRF model to obtain the label sequence with the highest probability value, and obtain the training and test entities, training and test attributes and training and test relationships; The relational knowledge graph is obtained by combining the training and testing entities, the training and testing attributes and the training and testing relationships.

4. The intelligent answering method based on training and testing multi-source heterogeneous knowledge according to claim 1 is characterized in that: Obtaining the training and test answer video includes: The set of image prompt texts are respectively input into a generator network for generating an image foreground and a generator network for generating an image background, and a Self-Attention mechanism is introduced to generate foreground and background text images; Using a discriminator network to discriminate the generated foreground and background text images and the real foreground images and background images, and generate the set of images corresponding to the image prompt text; Calculate similarity between the pictures corresponding to the set of picture prompt texts, and obtain target training and test videos from multi-source heterogeneous knowledge of training and test; The target training test video and the set of picture description voices are synthesized to obtain the training test answer video.

5. The intelligent answering method based on training and testing multi-source heterogeneous knowledge according to claim 4 is characterized in that: The training process of using the discriminator network to distinguish the generated foreground and background text images from the real foreground images and background images is as follows: Among them, x represents the training test image used for training, e represents the training test answer text, Q1 represents the training test foreground discriminator network, Q1(x) represents the discriminator output of the training test foreground real image, Q2 represents the training test background discriminator network, Q2(x) represents the discriminator output of the training test background real image, K1 represents the training test foreground generator network, K1(e) represents the training test foreground image generated by the model, K2 represents the training test background generator network, K2(e) represents the training test background image generated by the model, and E x～p(x) represents the expected distribution of training and test image sample data, E e～p(e) Represents the expectation of the distribution of sample data of training and test answers, through the joint Maximize the objective function, joint Minimize the objective function and get the optimal parameters.

Images