Method for evaluating and generating interpretable Chinese dialogue based on multi-modal perception

Abstract

The invention discloses an interpretable Chinese dialogue evaluation and generation method based on multi-modal perception, and the method comprises the steps: 1, collecting a current expression picture of a learner through a camera, and carrying out the emotion recognition of the learner based on a Faster RCNN network; 2, predicting the attention degree of the knowledge points, and extracting nodes of the attention degree in the knowledge graph to obtain a learning path of a learner; 3, taking the mastering condition of the learner on the knowledge point mark as a label, and performing classification; 4, classifying the current attributes of the learners; 5, calculating interpretable evaluation of the Chinese learner through an axiom fuzzy set membership function; step 6, realizing dialogue generation by using the pre-trained Chinese language model; and step 7, visualizing the multi-modal dialogue evaluation through a display. Personalized dialogue practice opportunities can be provided, the workload of teachers and assistant teaching is reduced, and practical teaching conditions are provided for learners.

Description

Technical Field

[0001] The invention relates to an interpretable Chinese dialogue evaluation and generation method based on multimodal perception, belonging to the technical field of artificial intelligence. Background Art

[0002] With the in-depth application of technologies such as artificial intelligence, big data and the Internet of Things, the intelligent development of national language education has ushered in a new direction. As one of the key tasks of smart education, the intelligent dialogue-assisted learning model is an important way to improve and test learners' comprehensive ability to use the national common language.

[0003] The existing intelligent dialogue and Chinese proficiency evaluation models for students' Chinese language learning mainly have the following problems: a. There are large differences in students' Chinese comprehension and expression abilities, and beginners often make grammatical and word order errors and inaccurate word choice in dialogues, which makes the dialogue data contain noise, affecting the content understanding and high-quality generation of the dialogue system; b. Existing evaluation models generally do not propose a universal modeling method, but focus on modeling the impact of specific factors on Chinese expression (such as learning environment, etc.); c. The current evaluation models are still relatively weak in interpretability and comprehensibility, and cannot provide teachers with intuitive teaching suggestions and concise personalized teaching plans. Summary of the invention

[0004] The present invention designs and develops an interpretable Chinese dialogue evaluation and generation method based on multimodal perception, which can overcome the problems that the existing Chinese dialogue teaching assistant model is difficult to accurately understand and answer the dialogue content of Chinese learners and the evaluation of learners' Chinese dialogue ability is unexplainable in complex educational application scenarios.

[0005] The technical solution provided by the present invention is:

[0006] A method for interpretable Chinese dialogue evaluation and generation based on multimodal perception, comprising:

[0007] Step 1: Collect photos of the learner's current expression through the camera, and perform learner emotion recognition based on the Faster RCNN network;

[0008] Step 2: Predict the attention of knowledge points, extract the nodes of attention in the knowledge graph, and obtain the learner's learning path;

[0009] Step 3: Input the learner’s learning path information and knowledge point entities as nodes of the graph structure into the graph convolutional neural network through the subject knowledge graph, encode the learner’s learned information in combination with the BiLSTM network, and use the learner’s mastery of the knowledge point tags as labels and classify them;

[0010] Step 4: Define a set of simple concepts based on the three semantic attributes of learning path, dialogue sentiment, and cognitive evaluation to classify the current attributes of the learner;

[0011] Step 5: Calculate the interpretable evaluation of Chinese learners through the axiomatic fuzzy set membership function;

[0012] Step 6: Input the multimodal feature vectors of the conversation context, expression, and learning path into the pre-trained BERT model to obtain feature fusion. Use the Transformer structure to fully learn the contextual relationship, obtain the fusion features of the conversation, and use the pre-trained Chinese language model to achieve conversation generation.

[0013] Step 7: Visualize the multimodal dialogue evaluation through the display.

[0014] Preferably, the step 2 comprises:

[0015] Extract the learner's attention record of the learned knowledge points in the learning record as a label;

[0016] Vectorize and embed learners’ online learning records based on the pre-trained BERT model;

[0017] After characterizing the knowledge nodes based on the RotatE model, the feature vector is obtained by graph embedding;

[0018] The vectorized embedding of the learning records and the graph embedding features of the knowledge nodes are concatenated and input into the long short-term memory network with a time dimension to complete the training of the knowledge point attention prediction model;

[0019] The attention of the unlearned knowledge points in the knowledge graph is predicted based on the attention prediction model, and the top 50% of the knowledge points are extracted to obtain the learner's learning path.

[0020] Preferably, in step 3, the set of simple concepts is:

[0021] M={m i,j 1≤i≤3,1≤j≤3};

[0022] In the formula, m i,1 Indicates that the learner is in the current attribute f i "High score", m i,2 Indicates that the learner is in the current attribute f i "Medium score", m i,3 Indicates that the learner is in the current attribute f i on "low score";

[0023] Complex concepts are obtained from simple concepts through the logical operations of "and (∧)" and "or (∨)".

[0024] Preferably, in step 5, the calculation formula of the axiomatic fuzzy set membership function is:

[0025]

[0026] Where X is the set of learners, M is the set of simple concepts on X, (M,τ,X) is the axiomatic fuzzy set structure on X, for a learner x∈X, and the set of simple concepts gather For the set A i For any simple concept m in , there is a set of learners y whose degree of belonging to concept m is not as great as that of learner x.

[0027] Preferably, in step 6, using a pre-trained Chinese language model to implement dialogue generation includes:

[0028] The probability that the i-th word starts the dialogue output is:

[0029]

[0030] The probability that the jth word ends the dialogue output is:

[0031]

[0032] Where S and H are additional learning parameter matrices, H i and H j are the BERT embedding vectors of the i-th and j-th words in H respectively;

[0033] H = BERT([CLS] + p t +[SEP]+p t-1 +l t +e t +c t +[SEP]);

[0034] In the formula, p t is the conversation content at the current time t, p t-1 is the historical dialogue at time t-1, l t is the learning path content at time t, e t is the learner’s emotion label content at time t, [CLS] and [SEP] are special identifiers of the pre-trained BERT model, indicating the beginning and end of the input, respectively, c t Cognitive label.

[0035] Preferably, in step seven, the multimodal resource data, learner observation dimension information, evaluation and analysis results, and personalized dialogue content are displayed on the web through the Gradio python library.

[0036] Beneficial effects of the present invention: The interpretable Chinese dialogue evaluation and generation method based on multimodal perception provided by the present invention is conducive to the integration of smart education big data resources and the improvement of the level of education informatization. With the help of multimodal perception, feature fusion of education big data is achieved, and related attributes are integrated into a graph structure, which is conducive to the integration of smart education big data resources and the improvement of the level of education informatization.

[0037] In view of the differences in learners' common language oral proficiency, an interpretable Chinese language proficiency evaluation model is more likely to gain the trust of users. The present invention provides a personalized national common language intelligent dialogue system to achieve precise education management.

[0038] The multimodal dialogue generation model takes into account the learner's emotions and learning habits in the current conversation, provides personalized dialogue practice opportunities, reduces the workload of teachers and teaching assistants, and provides learners with real-time and efficient intelligent teaching assistant conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The multimodal perception dialogue generation system framework described in the present invention;

[0040] Figure 2 The learning graph path prediction module based on long short-term memory network described in the present invention;

[0041] Figure 3 The learner cognitive assessment module based on graph convolutional neural network of the present invention;

[0042] Figure 4 The feature fusion dialogue generation model based on the pre-training model described in the present invention;

[0043] Figure 5 This is the visual intelligent Chinese learning evaluation interface described in the present invention. DETAILED DESCRIPTION

[0044] The present invention is further described in detail below in conjunction with the accompanying drawings so that those skilled in the art can implement the invention with reference to the description.

[0045] like Figure 1-5 As shown, the present invention provides an interpretable Chinese dialogue evaluation and generation method based on multimodal perception, proposes a high-quality dialogue generation model and an interpretable evaluation model suitable for the cultivation of learners' Chinese proficiency, and solves the problems that the existing Chinese dialogue teaching assistant model is difficult to accurately understand and answer the dialogue content of Chinese learners and the evaluation of learners' Chinese proficiency is unexplainable in complex educational application scenarios.

[0046] The present invention also provides an interpretable Chinese dialogue evaluation and generation model based on multimodal perception, including: a camera and a display, wherein the camera is used to collect the expressions of the interlocutors, and the display is used as a speaker for dialogue output and evaluation content display.

[0047] The present invention provides an interpretable Chinese dialogue evaluation and generation method based on multimodal perception, which adopts an interpretable Chinese dialogue evaluation and generation model based on multimodal perception, including:

[0048] Step 1: Learner emotion recognition based on Faster RCNN network

[0049] In view of the fact that there is little labeled data for learners' expression images, a small sample pseudo-label generation method with a memory module is designed for data expansion. The model vectorizes and embeds a small amount of labeled data, and calculates the cosine similarity between the encoding result and the category memory matrix in the memory module. Then the encoded unlabeled image is similar to each category memory matrix to obtain the pseudo label. The expanded labeled data is used to train the Faster RCNN network for learner emotion recognition, including:

[0050] The labeled sentiment data set is recorded as:

[0051] In the formula, It is a tagged expression picture. i yes emoticon tag;

[0052] The unlabeled data set is recorded as:

[0053] In the formula, Are there emoticon pictures with or without labels?

[0054] The initialized memory vector set is: V = {v1, v2, v3};

[0055] v1, v2, and v3 correspond to the three feature vectors of positive, neutral, and negative learning emotions, respectively;

[0056] Use the labeled dataset S to train an encoding network f, and update the memory module V by calculating the cosine similarity between the encoding vector and the memory vector. Take the category with the largest cosine similarity as the pseudo label.

[0057] Step 2: Learning graph path prediction based on long short-term memory network

[0058] As learners’ knowledge reserves become richer, the dialogue questions they raise may involve information fusion and reasoning of multiple knowledge points, so a knowledge retrieval model is needed to fully find the knowledge required for the dialogue. The learning content is marked in the knowledge graph based on the learners’ online learning check-in records. The learning materials are vectorized and embedded using the pre-trained BERT model, and the knowledge point nodes are represented by RotatE to obtain feature vectors from graph embedding. The above features are spliced ​​and input into the long-short memory network with a time dimension to predict the attention of knowledge points, such as Figure 2 Then, the attention degree of the knowledge points adjacent to the learned knowledge points is predicted, and the nodes with high attention degree are extracted from the knowledge graph to obtain the learner's learning path.

[0059] Step 3: Cognitive diagnosis based on graph convolutional neural network

[0060] During the self-learning process, learners will leave information about their learning paths and knowledge point exercises and test records, as well as self-scoring information about their knowledge mastery. The subject knowledge graph connects related knowledge in series, and can extract the learner's path from the graph. The associated entities are input into the graph convolutional neural network (GCN) as nodes of the graph structure, and the BiLSTM network is used to encode the learner's learned information. The learner's mastery of the knowledge point mark is used as a label, which is divided into three types: proficient mastery, understanding of basic concepts, and not mastered. Figure 3 The trained network structure can be used to further predict the learner's mastery of unlabeled knowledge points and realize the cognitive diagnosis of the learner's domain knowledge.

[0061] Step 4: Construction of semantic framework based on axiomatic fuzzy sets

[0062] As learners' knowledge reserves become richer, the proposed dialogue may involve information fusion and reasoning of multiple knowledge points, so the model needs to be able to comprehensively find the required knowledge for analysis. The project intends to implement interpretable learner ability evaluation based on the semantic analysis of learners and the axiomatic fuzzy set from the learner's characteristics. It defines three semantic attributes: learning path, dialogue emotion, and cognitive evaluation, which are used to describe learners from different dimensions. Define a set of simple concepts based on the three attributes M = {m i,j |1≤i≤3,1≤j≤3}, where m i,1 , m i,2 , m i,3 They represent the learner’s current attribute f i The complex concepts are obtained by logical operations of "and (∧)" and "or (∨)" from simple concepts.

[0063] Step 5: Explainable review generation

[0064] Axiomatic fuzzy sets propose a membership function based on the probability distribution of data features. The calculation formula of the axiomatic fuzzy set membership function is:

[0065]

[0066] Where X is the set of learners, M is the set of simple concepts on X, (M,τ,X) is the axiomatic fuzzy set structure on X, for a learner x∈X, and the set of simple concepts gather For the set A i For any simple concept m in , there is a set of learners y whose degree of belonging to concept m is not as great as that of learner x.

[0067] Step 6: Dialogue generation based on pre-trained Chinese language model

[0068] The multimodal feature vectors such as conversation context, expression, learning path, and learning style are jointly input into the pre-trained model BERT to obtain feature fusion. The Transformer structure is used to fully learn the contextual relationship to obtain the fusion features of the conversation, such as Figure 4 As shown in the figure. Use a large pre-trained Chinese language model to achieve dialogue generation, including:

[0069] The probability that the i-th word starts the dialogue output is:

[0070]

[0071] The probability that the jth word ends the dialogue output is:

[0072]

[0073] Where S and H are additional learning parameter matrices, H i and H j are the BERT embedding vectors of the i-th and j-th words in H respectively;

[0074] H = BERT([CLS] + p t +[SEP]+p t-1 +l t +e t +c t +[SEP]);

[0075] In the formula, p t is the conversation content at the current time t, p t-1 is the historical dialogue at time t-1, l t is the learning path content at time t, e tis the learner’s emotion label content at time t, [CLS] and [SEP] are special identifiers of the pre-trained BERT model, indicating the beginning and end of the input, respectively, c t is the cognitive label and e is the natural index.

[0076] Step 7: Visualize the multimodal dialogue evaluation through the display, such as Figure 5 shown.

[0077] In the present invention, as a preferred embodiment, the multimodal resource data, learner observation dimension information, evaluation analysis results and personalized dialogue content are displayed on the web through the Gradio python library.

[0078] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A method for evaluating and generating interpretable Chinese dialogue based on multimodal perception, characterized in that: include: Step 1: Collect photos of the learner's current expression through the camera, and perform learner emotion recognition based on the Faster RCNN network; Step 2: Predict the attention of knowledge points, extract the nodes of attention in the knowledge graph, and obtain the learner's learning path; Step 3: Input the learner’s learning path information and knowledge point entities as nodes of the graph structure into the graph convolutional neural network through the subject knowledge graph, encode the learner’s learned information in combination with the BiLSTM network, and use the learner’s mastery of the knowledge point tags as labels and classify them; Step 4: Define a set of simple concepts based on the three semantic attributes of learning path, dialogue sentiment, and cognitive evaluation to classify the current attributes of the learner; Step 5: Calculate the interpretable evaluation of Chinese learners through the axiomatic fuzzy set membership function; Step 6: Input the multimodal feature vectors of the conversation context, expression, and learning path into the pre-trained BERT model to obtain feature fusion. Use the Transformer structure to fully learn the contextual relationship, obtain the fusion features of the conversation, and use the pre-trained Chinese language model to achieve conversation generation. Step 7: Visualize the multimodal dialogue evaluation through the display.

2. The method for evaluating and generating interpretable Chinese dialogue based on multimodal perception according to claim 1 is characterized in that: The second step comprises: Extract the learner's attention record of the learned knowledge points in the learning record as a label; Vectorize and embed learners’ online learning records based on the pre-trained BERT model; After characterizing the knowledge nodes based on the RotatE model, the feature vector is obtained by graph embedding; The vectorized embedding of the learning records and the graph embedding features of the knowledge nodes are concatenated and input into the long short-term memory network with a time dimension to complete the training of the knowledge point attention prediction model; The attention of the unlearned knowledge points in the knowledge graph is predicted based on the attention prediction model, and the top 50% of the knowledge points are extracted to obtain the learner's learning path.

3. The method for evaluating and generating interpretable Chinese dialogue based on multimodal perception according to claim 2 is characterized in that: In step 3, the set of simple concepts is: M={m i,j 1≤i≤3,1≤j≤3}; In the formula, m i,1 Indicates that the learner is in the current attribute f i "High score", m i,2 Indicates that the learner is in the current attribute f i "Medium score", m i,3 Indicates that the learner is in the current attribute f i "Low score"; Complex concepts are obtained from simple concepts through the logical operations of "and (∧)" and "or (∨)".

4. The method for evaluating and generating interpretable Chinese dialogue based on multimodal perception according to claim 3 is characterized in that: In the step 5, the calculation formula of the axiomatic fuzzy set membership function is: Where X is the set of learners, M is the set of simple concepts on X, (M,τ,X) is the axiomatic fuzzy set structure on X, for a learner x∈X, and the set of simple concepts gather For the set A i For any simple concept m in , there is a set of learners y whose degree of belonging to concept m is not as great as that of learner x.

5. The method for evaluating and generating interpretable Chinese dialogue based on multimodal perception according to claim 4 is characterized in that: In step 6, the dialogue generation is realized by using the pre-trained Chinese language model, including: The probability that the i-th word starts the dialogue output is: The probability that the jth word ends the dialogue output is: Where S and H are additional learning parameter matrices, H i and H j are the BERT embedding vectors of the i-th and j-th words in H respectively; H BERT([CLS]+p t +[SEP]+p t-1 +l t +e t +c t +[SEP]) In the formula, p t is the conversation content at the current time t, p t-1 is the historical dialogue at time t-1, l t is the learning path content at time t, e t is the learner’s emotion label content at time t, [CLS] and [SEP] are special identifiers of the pre-trained BERT model, indicating the beginning and end of the input, respectively, c t is the cognitive label and e is the natural index.

6. The method for evaluating and generating interpretable Chinese dialogue based on multimodal perception according to claim 5 is characterized in that: In step seven, the multimodal resource data, learner observation dimension information, evaluation analysis results, and personalized dialogue content are displayed on the web through the Gradio python library.

Images