Cognitive diagnosis method and system based on multimodal complementary graph neural network

Abstract

The present application discloses a cognitive diagnosis method and system based on a multimodal complementary graph neural network. The method includes acquiring learning data, wherein the learning data includes video resource data, voice resource data, and exercise resource data; determining a multimodal resource complementary graph of the learning data based on the learning data; interactively processing the multimodal resource complementary graph to obtain aggregated embedded data of the learning data; inputting the aggregated embedded data into a cognitive diagnosis model to obtain a predicted result of cognitive diagnosis. The system includes an acquisition module, a multimodal resource complementary graph module, an interactive processing module, and a prediction module. By using the method in the present application, it is possible to fully extract multimodal learning data, improve the efficiency of the cognitive diagnosis model, and improve the accuracy and prediction effect of the model. The present application can be widely used in the field of neural network technology.

Description

Technical Field

[0001] The present application relates to the field of neural network technology, and in particular to a cognitive diagnosis method and system based on a multimodal complementary graph neural network. Background Art

[0002] Intelligent education has been widely promoted and used. In intelligent education, it is very important to grasp the knowledge status of each student and conduct a comprehensive teaching evaluation of students. In related technologies, cognitive diagnosis is used to evaluate students' cognitive processes. However, the accuracy and computational efficiency of cognitive diagnosis still need to be improved. Summary of the invention

[0003] The purpose of this application is to solve one of the technical problems existing in the prior art to at least a certain extent.

[0004] To this end, the purpose of the present invention is to provide an efficient cognitive diagnosis method and system based on multimodal complementary graph neural network.

[0005] Another object of an embodiment of the present application is to provide a cognitive diagnosis system based on a multimodal complementary graph neural network.

[0006] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present application include:

[0007] On the one hand, the embodiment of the present application provides a cognitive diagnosis method based on a multimodal complementary graph neural network, comprising the following steps:

[0008] The cognitive diagnosis method based on the multimodal complementary graph neural network of the embodiment of the present application includes obtaining learning data, wherein the learning data includes video resource data, voice resource data and exercise resource data; determining the multimodal resource complementary graph of the learning data according to the learning data; interactively processing the multimodal resource complementary graph to obtain aggregated embedded data of the learning data; inputting the aggregated embedded data into the cognitive diagnosis model to obtain the prediction result of cognitive diagnosis. By using the above method, the multimodal learning data can be fully extracted, the efficiency of the cognitive diagnosis model can be improved, and the accuracy and prediction effect of the model can be improved.

[0009] In addition, the cognitive diagnosis method based on the multimodal complementary graph neural network according to the above embodiment of the present application may also have the following additional technical features:

[0010] Furthermore, the cognitive diagnosis method based on a multimodal complementary graph neural network in an embodiment of the present application, wherein the multimodal resource complementary graph is interactively processed to obtain aggregated embedded data of the learning data, comprises the following steps: processing the multimodal resource complementary graph through an MLP to obtain a multimodal feature vector of the learning data; determining, based on the learning data, a concept relationship graph corresponding to the knowledge points of the learning data; and iteratively interactively processing the multimodal feature vector and the concept relationship graph to obtain aggregated embedded data of the learning data.

[0011] Furthermore, in one embodiment of the present application, after the step of iteratively interactively processing the multimodal feature vector and the concept relationship graph, the following steps are included: updating the node feature information through iterative interactive processing; processing the node feature information through an attention mechanism to obtain the weight of the node feature information; and obtaining the aggregated embedded data of the learning data based on the node feature information and the weight.

[0012] Further, in one embodiment of the present application, inputting the aggregated embedded data into a cognitive diagnosis model to obtain a prediction result of cognitive diagnosis comprises the following steps:

[0013] The aggregated embedded data and one-hot encoding are subjected to embedding fusion processing to obtain learning aggregated data;

[0014] The learning aggregation data is processed according to the following formula to obtain the model input data x:

[0015]

[0016] The learning aggregation data includes: speech difficulty vector F o , video difficulty vector F v , the difficulty vector F of the exercise diff , the discrimination vector F of the exercise disc , the importance vector F of the exercise import , solve the problem velocity vector F rapid , the student's proficiency vector F s ; In the formula Used to represent the exercise dependency aggregation embedding data; x is used to represent the model input data;

[0017] The model input data is input into a cognitive diagnosis model to obtain a prediction result of cognitive diagnosis.

[0018] Further, in one embodiment of the present application, the cognitive diagnosis model includes a bidirectional GRU neural network module, the GRU includes an update gate and a reset gate, and the step of inputting the model input data into the cognitive diagnosis model includes the following steps:

[0019] Inputting the model input data into a bidirectional GRU neural network module, and updating the node features of the model input data;

[0020] Among them, the update gate z t The formula is:

[0021] z t =σ(W 1 (z) x t +U 1 (z) h t-1 );

[0022] Reset Gate t The formula is:

[0023] r t =σ(W 2 (z) x t +U 2 (z) h t-1 );

[0024] x t is the input vector at time t, W (z) and U (z) is the weight matrix, h t-1 is the state at time t-1, and σ(·) is the activation function.

[0025] Furthermore, in one embodiment of the present application, determining a multimodal resource complementary graph of the learning data based on the learning data includes the following steps: processing the video resource data through a multimode Transformer encoder to obtain a video semantic feature vector; processing the speech resource data through an HMM to obtain a speech semantic feature vector; processing the practice resource data through a TIM Transformer to obtain a practice semantic feature vector; determining the correlation between different learning data based on the video semantic feature vector, the speech semantic feature vector and the practice semantic feature vector; if the correlation is less than a preset threshold, determining that a complementary relationship exists between the corresponding learning data; and determining the multimodal resource complementary graph of the learning data based on the complementary relationship.

[0026] Furthermore, in one embodiment of the present application, the correlation between the different learning data is obtained by the following formula:

[0027]

[0028] Among them, e′ i and e′ j Used to represent different learning data; Ki To characterize e′ i The number of resource data included, M k To characterize e′ i The semantic feature vector corresponding to the resource data in K j To characterize e′ j The number of resource data included, N k To characterize e′ j The semantic feature vector corresponding to the resource data in .

[0029] On the other hand, the embodiment of the present application proposes a cognitive diagnosis system based on a multimodal complementary graph neural network, comprising:

[0030] An acquisition module, used for acquiring learning data, wherein the learning data includes video resource data, voice resource data and exercise resource data;

[0031] A multimodal resource complementation graph module, used to determine a multimodal resource complementation graph of the learning data according to the learning data;

[0032] An interactive processing module, used for interactively processing the multimodal resource complementarity graph to obtain aggregated embedded data of the learning data;

[0033] A prediction module is used to input the aggregated embedded data into a cognitive diagnosis model to obtain a prediction result of cognitive diagnosis; wherein the aggregated embedded data is used to characterize the speech dependency, video dependency, exercise dependency and concept dependency of the learning data.

[0034] On the other hand, an embodiment of the present application provides a cognitive diagnosis device based on a multimodal complementary graph neural network, comprising:

[0035] at least one processor;

[0036] at least one memory for storing at least one program;

[0037] When the at least one program is executed by the at least one processor, the at least one processor implements any of the above-mentioned cognitive diagnosis methods based on multimodal complementary graph neural networks.

[0038] On the other hand, an embodiment of the present application provides a storage medium, which stores a program executable by a processor, and the program executable by the processor is used to implement any of the above-mentioned cognitive diagnosis methods based on multimodal complementary graph neural networks when executed by the processor.

[0039] The embodiment of the present application proposes a cognitive diagnosis method and system based on a multimodal complementary graph neural network, the method comprising acquiring learning data, wherein the learning data comprises video resource data, voice resource data and exercise resource data; determining a multimodal resource complementary graph of the learning data according to the learning data; interactively processing the multimodal resource complementary graph to obtain aggregated embedded data of the learning data; inputting the aggregated embedded data into a cognitive diagnosis model to obtain a predicted result of cognitive diagnosis. The system comprises an acquisition module, a multimodal resource complementary graph module, an interactive processing module and a prediction module. By using the method in the present application, it is possible to fully extract multimodal learning data, improve the efficiency of the cognitive diagnosis model, and improve the accuracy and prediction effect of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present application or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present application. For technical personnel in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 A schematic diagram of a flow chart of an embodiment of a cognitive diagnosis method based on a multimodal complementary graph neural network provided in the present application;

[0042] Figure 2 A schematic flow chart of another embodiment of a cognitive diagnosis method based on a multimodal complementary graph neural network provided in the present application;

[0043] Figure 3 A schematic diagram of the structure of an embodiment of a bidirectional GRU neural network module provided in this application;

[0044] Figure 4 A schematic diagram of the architecture of another embodiment of the bidirectional GRU neural network module provided in the present application;

[0045] Figure 5 A schematic diagram of a process for constructing a multimodal resource complementation graph according to an embodiment of the present application;

[0046] Figure 6 A schematic diagram of the structure of an embodiment of a cognitive diagnosis system based on a multimodal complementary graph neural network provided by the present application;

[0047] Figure 7 A schematic structural diagram of an embodiment of a cognitive diagnosis device based on a multimodal complementary graph neural network provided in the present application. DETAILED DESCRIPTION

[0048] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limitations on the present application. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0049] With the promotion and development of intelligent education, it is crucial to grasp the knowledge status of each student and conduct a comprehensive teaching evaluation of students. Most test technologies only focus on students' scores or answer results, and do not pay attention to students' internal cognitive structure and knowledge status. Students' cognitive processes cannot be analyzed and diagnosed. Cognitive diagnosis, as a technology of intelligent education, can effectively solve the above problems. The cognitive diagnosis method in related technologies uses manually designed functions, which makes it difficult to capture the complex relationship between students and exercises, and manually designing and implementing functional functions of the complex relationship between exercises and knowledge points often consumes a lot of manpower, energy and time.

[0050] The following describes in detail a cognitive diagnosis method and system based on a multimodal complementary graph neural network according to an embodiment of the present application with reference to the accompanying drawings. First, the cognitive diagnosis method based on a multimodal complementary graph neural network according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0051] Reference Figure 1 In an embodiment of the present application, a cognitive diagnosis method based on a multimodal complementary graph neural network is provided. The cognitive diagnosis method based on a multimodal complementary graph neural network in the embodiment of the present application can be applied to a terminal, can be applied to a server, can also be software running in a terminal or a server, etc. The terminal can be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The cognitive diagnosis method based on a multimodal complementary graph neural network in the embodiment of the present application mainly includes the following steps:

[0052] S110: Acquire learning data, where the learning data includes video resource data, voice resource data, and exercise resource data;

[0053] In this step, learning data is obtained. The learning data is used to characterize the learning data of students who need cognitive diagnosis. In some possible implementations, the learning data may be video resource data, the learning data may also be voice resource data, and the learning data may also be exercise resource data. By acquiring different types of learning data, the learning process of students can be fully explored. Exemplarily, the video resource data may be a teacher's lecture video or a related popular science video; the voice resource data may be data in the form of a teacher's lecture voice; the exercise resource data may be PPT data of related courses, etc. This application does not limit the specific content and specific form of expression of the learning data.

[0054] S120: Determine a multimodal resource complement graph of the learning data according to the learning data;

[0055] In this step, by constructing a multimodal resource complementation graph of learning data, the subsequent cognitive diagnosis process can be better completed. The student's learning data contains multiple modalities, and the number of modalities contained in each learning data is different. Through the multimodal resource complementation graph, different learning data can be interactively learned, the correlation between different learning data can be fully explored, the missing information between each other can be filled, and the semantic gap between different modalities can be eliminated. This is to facilitate subsequent data processing.

[0056] S130: interactively processing the multimodal resource complementarity graph to obtain aggregated embedded data of the learning data;

[0057] In this step, the multimodal resource complementary graph is interactively processed to fully extract the correlation and dependency between the learning data, which can improve the efficiency and accuracy of cognitive diagnosis. In some possible implementations, the concept relationship graph is extracted through learning data, and the concept relationship graph is interactively processed with the video resource data, voice resource data, and exercise resource data to extract corresponding features and obtain aggregated embedded data. Figure 2 The multimodal representation embedding stage in the figure shows an implementation method and is not a specific limitation to the process.

[0058] S140: Inputting the aggregated embedded data into a cognitive diagnosis model to obtain a prediction result of cognitive diagnosis; wherein the aggregated embedded data is used to characterize the speech dependency, video dependency, exercise dependency and concept dependency of the learning data.

[0059] In this step, the cognitive diagnosis model is used to evaluate the student's learning process and obtain the prediction results. In some possible implementations, the cognitive diagnosis model can be a model based on a graph neural network. The graph neural network model can effectively extract node features and complex relationships between nodes. Multimodal data can be used to more fully diagnose students' cognitive status by constructing a graph neural network. Figure 2 In the cognitive diagnosis result prediction stage, after updating the node representation based on the cognitive diagnosis model, the probability of students answering the questions correctly is studied and the output y is output. Specifically, in the model training stage, the loss function used is shown in Formula 1, where it is established based on the output y and the label r of the real data:

[0060] loss=-(r i )∑ i (r i logy i +(1-r i )log(1-y i )) Formula 1

[0061] Optionally, the cognitive diagnosis method based on a multimodal complementary graph neural network in the embodiment of the present application, interactively processing the multimodal resource complementary graph to obtain aggregated embedded data of the learning data, includes the following steps:

[0062] Processing the multimodal resource complementarity graph through MLP to obtain a multimodal feature vector of the learning data;

[0063] Determine, based on the learning data, a concept relationship graph corresponding to the knowledge points of the learning data;

[0064] The multimodal feature vector and the concept relationship graph are iteratively and interactively processed to obtain aggregated embedding data of the learning data.

[0065] In this step, the interactive processing of the multimodal resource complementation graph includes the following steps:

[0066] S1301, characterization of multimodal feature vectors. Different learning data are regarded as super-vertices. Based on further narrowing the semantic gap between different modalities / resource data in the super-vertices, the interaction and fusion of different modal information are realized. In some possible implementations, a multi-layer perceptron (MLP) is used to construct an encoder to implement single feature encoding for the multimodal resources inside the super-vertices, and obtain the multimodal feature vectors of the learning data, namely, the speech feature vector O, the video feature vector V, and the practice feature vector Q.

[0067] S1302: construct a concept relationship graph. In some possible implementations, knowledge point IDs are extracted from the text exercises, and the relationships between different knowledge points (similarity and prerequisite dependency relationships) are considered to construct a concept relationship graph structure containing directed connections and undirected connections.

[0068] S1303, construct a multimodal interaction graph. Since there are connections between the knowledge points themselves and the exercises, voice, video and knowledge points, the interaction data can be described as a knowledge point similarity and dependency graph. Exercise Correlation Graph Video Correlation Graph and speech correlation graph Where C = {c 1 ,c 2 ,...,c K}, represents the set of knowledge concept nodes, and K represents the number of knowledge point concepts. represents multiple dependencies between knowledge point concepts, r represents a specific type of dependency (such as similarity and prerequisite), and each edge represents the relationship between different knowledge point concepts. 1 ,q 2 ,...,q M}, represents the training feature, M represents the number of training, Represents the correlation between the exercise and the knowledge point concept. 1 ,v 2 ,...,v T}, represents the video features, and T represents the number of videos. Represents the correlation between the video and the knowledge point concept. 1 ,o 2 ,...,o N}, represents the voice feature, Indicates the correlation between speech and knowledge point concepts.

[0069] S1304, multimodal feature vector interacts with graph structure. The speech feature vector, video feature vector, and exercise feature vector interact with the concept relationship graph respectively, and the multimodal relationship graph is depicted as follows: R vc ∪R oc ),in addition, and They represent the aggregate embedding of knowledge point concept dependency, exercise relevance, video relevance, and speech relevance, respectively, and ι represents the number of iterations.

[0070] See the above process diagram for Figure 2In the multimodal representation embedding stage, through the interactive process, the information in the learning data is fully extracted to improve the efficiency and accuracy of cognitive diagnosis.

[0071] Optionally, the cognitive diagnosis method based on a multimodal complementary graph neural network in the embodiment of the present application, after the step of iteratively interactively processing the multimodal feature vector and the concept relationship graph, comprises the following steps:

[0072] Update node feature information through iterative interactive processing;

[0073] Processing the node feature information through an attention mechanism to obtain a weight of the node feature information;

[0074] According to the node feature information and the weight, aggregated embedding data of the learning data is obtained.

[0075] In this step, the node feature information is updated through iterative interaction processing; the updated node feature information is processed by the attention mechanism to obtain the weight of the node feature information; based on the node feature information and the weight, the aggregated embedded data of the learning data is obtained. In some possible implementations, the process of fully expressing the speech features, video features, exercise features and knowledge points is as follows: In the concept relationship diagram module, for The concept C is transferred using the graph neural network to obtain the similarity and dependency features between C. After ι iterative updates, the node-level attention is used to obtain the weights of the node features and edge features to obtain the aggregate embedding of the knowledge point concept dependency. In the multimodal feature vector, namely the speech vector, video vector, exercise vector and concept relationship graph interaction module, for The speech feature o, video feature v, and exercise feature q are interacted with concept C respectively. After ι iterations, the node features are continuously updated, and the node feature weights are obtained using the node-level attention mechanism. Finally, the aggregate embedding of speech relevance is obtained. Aggregate embedding of video relevance Aggregate embedding of exercise relevance in, and It is obtained through continuous optimization training.

[0076] Through the interaction of multimodal feature vectors, aggregated embedded data is obtained to improve the efficiency and accuracy of cognitive diagnosis.

[0077] Optionally, the cognitive diagnosis method based on a multimodal complementary graph neural network in the embodiment of the present application, wherein the aggregated embedded data is input into a cognitive diagnosis model to obtain a prediction result of cognitive diagnosis, comprises the following steps:

[0078] The aggregated embedded data and one-hot encoding are subjected to embedding fusion processing to obtain learning aggregated data;

[0079] The learning aggregation data is processed according to the following formula 2 to obtain model input data:

[0080]

[0081] The learning aggregation data includes: speech difficulty vector F o , video difficulty vector F v , the difficulty vector F of the exercise diff , the discrimination vector F of the exercise disc , the importance vector F of the exercise import , solve the problem velocity vector F rapid , the student's proficiency vector F s ;

[0082] The model input data is input into a cognitive diagnosis model to obtain a prediction result of cognitive diagnosis.

[0083] In this step, step S140 may include the following steps:

[0084] S1401, see Figure 2 The flowchart of the information aggregation and update phase in One-hot encoding of speech x o Embedding fusion to obtain the speech difficulty vector F ° ,Will One-hot encoding of the video x v Embedding fusion obtains the video difficulty vector F v ,Will One-hot encoding x of the exercise q Embedding fusion obtains the difficulty vector F of the exercise diff , the discrimination vector F of the exercise disc , the importance vector F of the exercise import 、Solve the velocity vector F rapid , encode the student's one-hot code x s Embed the student's proficiency vector F s The formulas involved in this step are Formula 3 to Formula 9:

[0085] F s =sigmoid(x s ×B) Formula 3

[0086]

[0087] F disc=sigmoid(x e ×D) Formula 5

[0088] F import =sigmoid(x e ×I) Formula 6

[0089] F r =sigmoid(x e ×T) Formula 7

[0090]

[0091]

[0092] Wherein B, D, I, T are trainable weight parameter matrices, and sigmoid is an activation function. It can be understood by those skilled in the art that the above are illustrative examples and are not intended to be specific limitations on trainable matrices and activation functions.

[0093] S1402, embedding exercise dependency aggregation With F o 、F v 、F diff 、F disc 、F import 、F rapid 、F s The dot product is performed to obtain the input x of the cognitive diagnosis model. See Formula 2.

[0094] By processing the learning aggregate data representing different learning states or processes through Formula 2 and inputting them into the cognitive diagnosis model, the accuracy of cognitive diagnosis can be improved by fully considering the students' learning data and increasing the input information.

[0095] Optionally, in the cognitive diagnosis method based on a multimodal complementary graph neural network in an embodiment of the present application, the cognitive diagnosis model includes a bidirectional GRU neural network module, the GRU includes an update gate and a reset gate, and the step of inputting the model input data into the cognitive diagnosis model includes the following steps:

[0096] Inputting the model input data into a bidirectional GRU neural network module, and updating the node features of the model input data;

[0097] Among them, the update gate z t The formula is shown in Formula 10:

[0098] z t =σ(W 1 (z) x t +U 1 (z) h t-1); Formula 10

[0099] Reset Gate t The formula for is shown in Formula 11:

[0100] r t =σ(W 2 (z) x t +U 2 (z) h t-1 ); Formula 11

[0101] x t is the input vector at time t, W (z) and U (z) is the weight matrix, h t-1 is the state at time t-1, and σ(·) is the activation function.

[0102] In this step, the model input data is input into the cognitive diagnosis model. In some possible implementations, the cognitive diagnosis model includes a bidirectional GRU neural network module, and the model input data is updated with node features inside the bidirectional GRU. The gated recurrent unit GRU includes two gated units, namely, update gate z t and reset gate r t , update the gate control information transmission, reset the gate control information forgetting. Specifically, see Figure 3 As shown, the process is:

[0103] At time t, calculate the update gate z t The formula is shown in formula 10,

[0104] Among them, x t is the input vector at time t, W (z) is the weight matrix, h t-1 is the state at time t-1, and the weight matrix U (z) Multiply them together for linear transformation, and after the activation function σ, z t The threshold is [0, 1].

[0105] At time t, calculate the reset gate r t The formula is shown in formula 11. Like the update gate, after a series of calculations, r t The threshold is [0, 1].

[0106] The information passes through the reset gate and the update gate to obtain the current memory content. The calculation formula is shown in Formula 12:

[0107] h' t =tanh(Wx t +r t ⊙Uh t-1) Formula 12

[0108] Finally, the current memory h' is collected by updating the gate t and the memory content of the previous moment h t-1 The required information finally yields h t And pass it to the next unit. Its calculation formula is shown in Formula 13:

[0109] h t =z t ⊙h' t +(1-z t )⊙h t-1 Formula 13

[0110] The vector H of the hidden layer in the bidirectional gated recurrent unit GRU is divided into forward and backward, and h represents the state of the hidden layer at each moment. The architecture of the bidirectional gated recurrent unit GRU is as follows: Figure 4 As shown, the relevant expressions refer to Formula 14 and Formula 15:

[0111]

[0112]

[0113] Therefore, the vector H of the hidden layer can connect the previous term and the next term in series, see formula 16:

[0114] H={h 1 ,h 2 , ..., h T} Formula 16

[0115] Considering that there is only one decoder, the two context vectors are concatenated together to form a vector z, which is passed through a linear layer g and an activation function tanh as the initial hidden state s of the decoder. 0 The calculation formula is as follows: Formula 17 to Formula 19:

[0116]

[0117]

[0118] z=s 0 Formula 19

[0119] Through the processing of the bidirectional GRU graph neural network module, the interactive information between learning data can be fully extracted, the computing efficiency can be improved, and the accuracy and prediction effect of the model can be improved.

[0120] Optionally, the cognitive diagnosis method based on a multimodal complementary graph neural network in the embodiment of the present application, wherein determining a multimodal resource complementary graph of the learning data according to the learning data, comprises the following steps:

[0121] Processing the video resource data through a multi-mode Transformer encoder to obtain a video semantic feature vector;

[0122] Processing the speech resource data through HMM to obtain a speech semantic feature vector;

[0123] Processing the exercise resource data through TIM Transformer to obtain an exercise semantic feature vector;

[0124] Determining correlations between different learning data according to the video semantic feature vector, the speech semantic feature vector, and the practice semantic feature vector;

[0125] If the correlation is less than a preset threshold, it is determined that there is a complementary relationship between the corresponding learning data;

[0126] According to the complementary relationship, a multimodal resource complementary graph of the learning data is determined.

[0127] Optionally, in the cognitive diagnosis method based on a multimodal complementary graph neural network in an embodiment of the present application, the correlation between the different learning data is obtained by the following formula 20:

[0128]

[0129] Among them, e′ i and e′ j Used to represent different learning data; K i To characterize e′ i The number of resource data included, M k To characterize e′ i The semantic feature vector corresponding to the resource data in K j To characterize e′ j The number of resource data included, N k To characterize e′ j The semantic feature vector corresponding to the resource data in .

[0130] In the above steps, the learning data is usually multimodal student learning data. Since each student will learn from multiple learning resources, and these learning resources can include three modes (voice, video, text), two of these modes, or one of these modes, in order to consider the importance of different modal information in the same resource for mining the relationship between resources and eliminate the semantic gap, each modal information in each learning resource is represented and learned, that is, "voice, video and text modalities are encoded using Hidden Markov Model (HMM), multi-mode Transformer encoder, and TIM Transformer model respectively" to form their respective representation vectors O', V', and Q'. The formula can be simply expressed as Formula 21 to Formula 23:

[0131] O'=HMM(audio) Formula 21

[0132] V'=Multi-mode Transformer(video) Formula 22

[0133] Q'=TIM Transformer(text) Formula 23

[0134] After obtaining the respective abstract semantic feature vectors, see Figure 5 , different learning data are represented by resource 1, resource 2, resource 3 and resource 4. Since the knowledge points that do not exist in resource 1 exist in resource 2 and resource 3, resource 1 has a complementary relationship with resource 2 and resource 3. Considering that there is an information complementary relationship between different resources, the interaction of these resources can make the multimodal data vector representation more sufficient and improve the student learning resource system. Therefore, different resources are composed and a threshold is set to judge the complementary relationship between different resources. Each resource corresponds to a super vertex, which contains the semantic representation vector of a single or multiple modes of the resource. The edges between resources represent complementary relationships. Each student has to learn multiple resources. Figure 5 Take a student learning four resources as an example. Specifically, the edge connection between super vertices represents the complementary relationship. By calculating the cosine similarity between different super vertices, the edges between different super vertices are constructed according to the preset threshold. The threshold can be used as a hyperparameter in actual experiments. The similarity relationship calculation formula between different super vertices can be expressed by Formula 20.

[0135] The adjacency matrix of the complementary relationship between super-vertices, that is, the multimodal resource complementarity graph, can be expressed by the following formula 24:

[0136]

[0137] Where E′ represents the super vertex set, Represents the super vertex e′ i and e′ j The weight of the relationship between i and e′ j There is complementarity between them. is 1 if the value is set to 0, otherwise it is 0.

[0138] In summary, this application proposes a cognitive diagnosis method based on a multimodal complementary graph neural network, which can fully extract multimodal learning data, improve the efficiency of the cognitive diagnosis model, and improve the accuracy and prediction effect of the model.

[0139] Secondly, refer to the attached Figure 6 A cognitive diagnosis system based on a multimodal complementary graph neural network is described according to an embodiment of the present application.

[0140] Figure 6 : is a schematic diagram of the structure of a cognitive diagnosis system based on a multimodal complementary graph neural network according to an embodiment of the present application, the system specifically includes:

[0141] An acquisition module 610 is used to acquire learning data, wherein the learning data includes video resource data, voice resource data and exercise resource data;

[0142] A multimodal resource complementation graph module 620, configured to determine a multimodal resource complementation graph of the learning data according to the learning data;

[0143] An interactive processing module 630, configured to interactively process the multimodal resource complementarity graph to obtain aggregated embedded data of the learning data;

[0144] The prediction module 640 is used to input the aggregated embedded data into the cognitive diagnosis model to obtain the prediction result of cognitive diagnosis; wherein the aggregated embedded data is used to characterize the speech dependency, video dependency, exercise dependency and concept dependency of the learning data.

[0145] It can be seen that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0146] Reference Figure 7 , the embodiment of the present application provides a cognitive diagnosis device based on a multimodal complementary graph neural network, comprising:

[0147] at least one processor 710;

[0148] At least one memory 720, used to store at least one program;

[0149] When the at least one program is executed by the at least one processor 710, the at least one processor 710 implements the cognitive diagnosis method based on the multimodal complementary graph neural network.

[0150] Similarly, the contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0151] The embodiment of the present application further provides a computer-readable storage medium, in which a program executable by the processor 710 is stored. The program executable by the processor 710 is used to execute the above-mentioned control method of the fresh air system when executed by the processor 710.

[0152] Similarly, the contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0153] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are expected, wherein the order of various operations is changed and the sub-operation described as a part of a larger operation is performed independently.

[0154] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional techniques of the engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0155] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several programs to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0156] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable programs for implementing the logical functions, and may be embodied in any computer-readable medium for use by a program execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch and execute a program from a program execution system, device or apparatus), or in conjunction with such program execution systems, devices or apparatuses. For purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by a program execution system, device or apparatus, or in conjunction with such program execution systems, devices or apparatuses.

[0157] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0158] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable program execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0159] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0160] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0161] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the described embodiments. Technical personnel familiar with the field may make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A cognitive diagnosis method based on multimodal complementary graph neural network, characterized in that: The following steps are involved: Acquire learning data, wherein the learning data includes video resource data, voice resource data and exercise resource data; Determining a multimodal resource complementation graph of the learning data according to the learning data; Interactively processing the multimodal resource complementarity graph to obtain aggregated embedding data of the learning data; Inputting the aggregated embedded data into a cognitive diagnosis model to obtain a prediction result of cognitive diagnosis; wherein the aggregated embedded data is used to characterize the aggregated embedded data of speech dependency, video dependency, exercise dependency and concept dependency of the learning data; The interactive processing of the multimodal resource complementarity graph to obtain aggregated embedded data of the learning data comprises the following steps: Processing the multimodal resource complementarity graph through MLP to obtain a multimodal feature vector of the learning data; Determine, according to the learning data, a concept relationship graph corresponding to the knowledge points of the learning data; the concept relationship graph structure includes directed connections and undirected connections; Iteratively and interactively processing the multimodal feature vector and the concept relationship graph to obtain aggregated embedding data of the learning data; Determining a multimodal resource complementation graph of the learning data according to the learning data comprises the following steps: Processing the video resource data through a multi-mode Transformer encoder to obtain a video semantic feature vector; Processing the speech resource data through HMM to obtain a speech semantic feature vector; Processing the exercise resource data through TIM Transformer to obtain an exercise semantic feature vector; Determining correlations between different learning data according to the video semantic feature vector, the speech semantic feature vector, and the practice semantic feature vector; If the correlation is less than a preset threshold, it is determined that there is a complementary relationship between the corresponding learning data; Determining a multimodal resource complementarity graph of the learning data according to the complementary relationship; The correlation between the different learning data is obtained by the following formula: Among them, e′ i and e′ j Used to represent different learning data; K i To characterize e′ i The number of resource data included, M k To characterize e′ i The semantic feature vector corresponding to the resource data in K j To characterize e′ j The number of resource data included, N k To characterize e′ i The semantic feature vector corresponding to the resource data in .

2. The cognitive diagnosis method based on multimodal complementary graph neural network according to claim 1 is characterized in that: After the step of iteratively interacting the multimodal feature vector with the concept relationship graph, the following steps are included: Update node feature information through iterative interactive processing; The node feature information is processed through an attention mechanism to obtain a weight of the node feature information; and based on the node feature information and the weight, aggregated embedding data of the learning data is obtained.

3. The cognitive diagnosis method based on multimodal complementary graph neural network according to claim 1 is characterized in that: The step of inputting the aggregated embedded data into a cognitive diagnosis model to obtain a prediction result of cognitive diagnosis comprises the following steps: The aggregated embedded data and one-hot encoding are subjected to embedding fusion processing to obtain learning aggregated data; The learning aggregation data is processed according to the following formula to obtain the model input data: The learning aggregation data includes: speech difficulty vector F o , video difficulty vector F v , the difficulty vector F of the exercise diff , the discrimination vector F of the exercise disc , the importance vector F of the exercise import , solve the problem velocity vector F rapid , the student's proficiency vector F s ; In the formula Used to represent the exercise dependency aggregation embedding data; x is used to represent the model input data; The model input data is input into a cognitive diagnosis model to obtain a prediction result of cognitive diagnosis.

4. The cognitive diagnosis method based on multimodal complementary graph neural network according to claim 3 is characterized in that: The cognitive diagnosis model includes a bidirectional GRU neural network module, the GRU includes an update gate and a reset gate, and the step of inputting the model input data into the cognitive diagnosis model includes the following steps: Inputting the model input data into a bidirectional GRU neural network module, and updating the node features of the model input data; Among them, the update gate z t The formula is: z t =′(W1 (z) x t +U1 (z) h t-1 ); Reset Gate t The formula is: r t =σ(W2 (z) x t +U2 (z) h t-1 ); x t is the input vector at time t, W (z) and U (z) is the weight matrix, h t-1 is the state at time t-1, and σ(·) is the activation function.

5. A cognitive diagnosis system based on multimodal complementary graph neural network, characterized in that: include: An acquisition module, used for acquiring learning data, wherein the learning data includes video resource data, voice resource data and exercise resource data; A multimodal resource complementation graph module, used to determine a multimodal resource complementation graph of the learning data according to the learning data; An interactive processing module, used for interactively processing the multimodal resource complementarity graph to obtain aggregated embedded data of the learning data; A prediction module, used to input the aggregated embedded data into a cognitive diagnosis model to obtain a prediction result of cognitive diagnosis; wherein the aggregated embedded data is used to characterize the aggregated embedded data of speech dependency, video dependency, exercise dependency and concept dependency of the learning data; Interaction processing module, used to: Processing the multimodal resource complementarity graph through MLP to obtain a multimodal feature vector of the learning data; Determine, according to the learning data, a concept relationship graph corresponding to the knowledge points of the learning data; the concept relationship graph structure includes directed connections and undirected connections; Iteratively and interactively processing the multimodal feature vector and the concept relationship graph to obtain aggregated embedding data of the learning data; Multimodal resource complementation graph module, used for: Processing the video resource data through a multi-mode Transformer encoder to obtain a video semantic feature vector; Processing the speech resource data through HMM to obtain a speech semantic feature vector; Processing the exercise resource data through TIM Transformer to obtain an exercise semantic feature vector; Determining correlations between different learning data according to the video semantic feature vector, the speech semantic feature vector, and the practice semantic feature vector; If the correlation is less than a preset threshold, it is determined that there is a complementary relationship between the corresponding learning data; Determining a multimodal resource complementarity graph of the learning data according to the complementary relationship; The correlation between the different learning data is obtained by the following formula: Among them, e′ i and e′ j Used to represent different learning data; K i To characterize e′ i The number of resource data included, M k To characterize e′ i The semantic feature vector corresponding to the resource data in K j To characterize e′ j The number of resource data included, N k To characterize e′ j The semantic feature vector corresponding to the resource data in .

6. A cognitive diagnosis device based on a multimodal complementary graph neural network, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the cognitive diagnosis method based on a multimodal complementary graph neural network as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to implement the cognitive diagnosis method based on a multimodal complementary graph neural network as described in any one of claims 1 to 4 when executed by the processor.

Images