Data and knowledge collaborative driving cognitive validity calculation model enhancement method and system

Abstract

The application discloses a data and knowledge collaborative driving cognitive validity calculation model enhancement method and system, and belongs to the field of deep learning.The method comprises the following steps: 1) obtaining teaching interpersonal interaction data; 2) obtaining a multi-modal interaction fusion hypergraph representation through a multi-Gaussian kernel Fourier transform mapping; 3) realizing effective extraction of multi-modal features through a hypergraph attention convolution; 4) constructing a priori knowledge gradient matrix and a Jocobian matrix of a cognitive validity calculation model; 5) introducing a priori knowledge regularization term to guide and constrain the training process of the cognitive validity calculation model; and 6) deploying the cognitive validity calculation model on a teaching application platform to realize real-time evaluation of teacher-student interaction validity, provide targeted teaching optimization and learning intervention strategies, and realize intelligent teaching optimization.The application integrates the knowledge in the cognitive field as priori into the cognitive validity calculation model in a regularized manner, and promotes the mapping relationship of learning to match the domain knowledge.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of deep learning, and particularly relates to a data and knowledge collaborative driving cognitive validity calculation model enhancement method and system. BACKGROUND

[0002] Teaching interpersonal interaction is a cooperative behavior aiming at knowledge construction and promoting high-order cognitive development. The "cognitive black box" in this process has hindered the development of high-quality teaching and learning. Researching the implicit cognitive mechanism of teaching interpersonal interaction is a multidisciplinary challenge to uncover the black box. The cognitive validity of teaching interpersonal interaction is an important reference and basis for reflecting cognitive ability, evaluating comprehensive effect, verifying mechanism effectiveness and discovering cognitive law. Generally speaking, the data in the process of teaching interpersonal interaction contains rich information about external and implicit cognitive processes. These data present a complex hypergraph structure with multiple sources and heterogeneous characteristics, complex relationships, high coupling degree and multiple modal relationships, forming a complex hypergraph.

[0003] The existing method uses feature analysis and multi-modal model to model the cognitive validity model. However, the existing data-driven cognitive validity calculation model has poor generalization, poor interpretability and data dependence.

[0004] In view of the problems existing in the cognitive validity calculation model, it is of great significance to realize the efficient collaboration of domain knowledge and teaching interpersonal interaction observation data, and to build a data-efficient processing and generalizable deep interactive cognitive validity calculation model. This will help to improve the performance, generalization and interpretability of the model decision, so as to better understand and optimize the cognitive validity of teaching interpersonal interaction. SUMMARY

[0005] The purpose of the present application is to solve the problems existing in the prior art and provide a data and knowledge collaborative driving cognitive validity calculation model enhancement method and system. The method of the present application aims at the limitations of poor generalization and data dependence of data-driven cognitive validity calculation model. On the basis of extracting features from interactive hypergraph data to predict cognitive validity, the cognitive field related knowledge (conclusions obtained by cognitive psychology) is taken as a priori and is integrated into the cognitive validity calculation model in a regularization manner to promote the mapping relationship learned by the cognitive validity calculation model to match the domain knowledge.

[0006] In order to achieve the above-mentioned application purposes, the present application specifically adopts the following technical solutions:

[0007] In a first aspect, the present application provides a data and knowledge collaborative driving cognitive validity calculation model enhancement method, which comprises the following steps:

[0008] S1, in a teaching interpersonal interaction scenario, obtain behavior data and brain neural data of students, extract key features from the two kinds of data, and construct a behavior correlation matrix and a brain neural correlation matrix respectively, which correspond to a behavior hypergraph representation and a brain neural hypergraph representation;

[0009] S2, using a multi-core learning method, performing Fourier transform on a Gaussian kernel function for explicitly calculating a mapping function, and calculating the behavior hypergraph representation and the brain neural hypergraph representation with the mapping function respectively, so as to map the behavior hypergraph representation and the brain neural hypergraph representation to a common high-dimensional feature space, obtaining an initial interactive fusion hypergraph representation, using a mixed L 21 norm constraint optimization on the kernel weight of each Gaussian kernel function, and obtaining an optimized weighted Gaussian kernel function by weighted combination of the Gaussian kernel function after the mixed L 21 norm constraint, performing Fourier transform on the optimized weighted Gaussian kernel function to obtain a new mapping function, and calculating the initial interactive fusion hypergraph representation with the new mapping function to obtain a multi-modal interactive fusion hypergraph representation;

[0010] S3, training the constructed cognitive validity calculation model, in the training process of the cognitive validity calculation model, obtaining an interactive feature representation by supergraph attention convolution operation of the cognitive validity calculation model on the multi-modal interactive fusion hypergraph representation, and predicting cognitive achievements in teaching interpersonal interaction by a classification function of the cognitive validity calculation model on the interactive feature representation; solving the partial derivative of the classification function on the interactive feature representation to obtain a Jocobian matrix, constructing a gradient matrix of prior knowledge by taking relevant knowledge in the cognitive field as priori, constructing a priori knowledge regularization term by using the gradient matrix of prior knowledge and the Jocobian matrix, adding a cross-entropy loss function and a weighted priori knowledge regularization term as a total loss function of the cognitive validity calculation model, updating the parameters of the cognitive validity calculation model based on minimizing the total loss function, and iteratively training until the total loss function converges, and finally obtaining the trained cognitive validity calculation model;

[0011] S4, deploying the trained cognitive validity calculation model to a teaching application platform supporting interpersonal interaction, obtaining interactive data between teachers and students in real time, inputting the interactive data between teachers and students into the trained cognitive validity calculation model to evaluate the cognitive validity of teaching interaction, and generating targeted teaching optimization and learning intervention strategies for the teaching application according to the cognitive validity result output by the cognitive validity calculation model, and providing auxiliary support for learning subjects in teaching and learning; wherein, the interactive data between teachers and students is the behavior data and brain neural data of students and the behavior data and brain neural data of teachers.

[0012] On the basis of the above scheme, each step can be implemented in the following preferred specific manner.

[0013] As a preferred embodiment of the first aspect, the implementation of step S1 comprises:

[0014] S11, obtaining the behavior data of the student by using a video recording system and a motion capture technology, wherein the behavior data includes the body movement and expression of the student; extracting the key point position of the body movement and the expression feature from the behavior data obtained by the video recording system by using a computer vision algorithm, and extracting the dynamic feature from the behavior data obtained by the motion capture technology; taking the key point position of the body movement and the expression feature and the dynamic feature as the key features of the behavior data;

[0015] S12, obtaining the brain neural data of the student by using an electroencephalogram and a functional magnetic resonance imaging technology, extracting the features from the brain neural data of the student by performing frequency analysis and correlation analysis to obtain the key features of the brain neural data, wherein the key features of the brain neural data include the frequency, amplitude and neural connectivity features of the brain neural data;

[0016] S13, taking the key features of the behavior data extracted in step S11 as the vertices in the behavior hypergraph, constructing the hyperedges in the behavior hypergraph according to the correlation between the behavior data, taking the key features of the brain neural data extracted in step S12 as the vertices in the brain neural hypergraph, and constructing the hyperedges in the brain neural hypergraph according to the correlation between the brain neural data;

[0017] S14, generating a behavior correlation matrix to depict the relationship between the vertices and the hyperedges in the behavior hypergraph, taking the behavior correlation matrix as the representation of the behavior hypergraph, and generating a brain neural correlation matrix to depict the relationship between the vertices and the hyperedges in the brain neural hypergraph, taking the brain neural correlation matrix as the representation of the brain neural hypergraph.

[0018] As a preferred embodiment of the first aspect, the implementation of step S2 comprises:

[0019] S21, selecting a Gaussian kernel function as a basic kernel function to process the nonlinear characteristics of the behavior hypergraph representation and the brain neural hypergraph representation; for two data points from the same hypergraph representation, the Gaussian kernel function is defined as:

[0020]

[0021] wherein μ is a kernel width parameter for controlling the smoothness of the Gaussian kernel function; K(x, y) represents the Gaussian kernel function of the two data points from the same hypergraph representation; x and y represent the two data points from the same hypergraph representation;

[0022] S22, applying Fourier transform on each Gaussian kernel function, calculating the mapping function explicitly, calculating the behavior hypergraph representation and the brain neural hypergraph representation respectively with the mapping function, so as to map the behavior hypergraph representation and the brain neural hypergraph representation to a common high-dimensional feature space, and obtaining an initial interactive fusion hypergraph representation;

[0023] For the Gaussian kernel function K(x,·), the mapping function is defined as:

[0024]

[0025] wherein K(x,·) represents the Gaussian kernel function of two data points from different hypergraph representations; FFT(·) represents Fourier transform;

[0026] S23, using mixed L 21 norm constraint to optimize the weight of each Gaussian kernel function, mixed L 21 norm constraint is defined as:

[0027]

[0028] wherein W i represents the weight matrix of the i-th Gaussian kernel function, A is a design matrix, B is a target matrix, and λ is a regularization parameter; W represents the weight matrix of all Gaussian kernel functions; represents the square of L2 norm;

[0029] S24, applying mixed L 21 norm constraint to the Gaussian kernel function to obtain an optimized weighted Gaussian kernel function K combined :

[0030]

[0031] wherein w i represents the weight of the i-th Gaussian kernel function after applying mixed L 21 norm constraint; K i(x,y) represents the i-th Gaussian kernel function after applying mixed L 21 norm constraint; and n represents the number of Gaussian kernel functions.

[0032] S25, applying Fourier transform on the optimized weighted Gaussian kernel function to obtain a new mapping function, calculating the initial interactive fusion hypergraph representation with the new mapping function, and obtaining a multi-modal interactive fusion hypergraph representation.

[0033] As a preferred embodiment of the above first aspect, in step S2, after applying mixed L 21 norm constraint, the weight of the i-th Gaussian kernel function is specifically:

[0034]

[0035] where β represents a learnable parameter; d i ,d j respectively represent the discriminative index of the i,jth Gaussian kernel function.

[0036] As a preferred embodiment of the first aspect, in step S3, the hypergraph attention convolution of the lth layer can be represented as:

[0037]

[0038] where X (l) and X (l+1) respectively represent the input feature and the output feature of the lth hypergraph attention convolution layer; σ is a nonlinear activation function; D and B respectively represent the degree matrix of the hypergraph representation of the interactive fusion of multiple modalities and the hyperedge; H represents the association matrix of the hyperedge to the vertex of the hypergraph representation of the interactive fusion of multiple modalities; W (l) represents the weight matrix of the lth hypergraph attention convolution layer; P (l) is the parameter matrix of the lth hypergraph attention convolution layer; T represents the matrix transpose; -1 represents the inverse matrix.

[0039] As a preferred embodiment of the first aspect, in step S3, the specific way of constructing the gradient matrix of the prior knowledge is:

[0040] S31, taking the knowledge related to the cognitive field as the prior knowledge, using the utility function U(x t ) to represent the high and low of the cognitive effect, for converting the form of the prior knowledge into mathematical expression;

[0041] S32, calculating the partial derivative of the utility function U(x t ) with respect to the interactive feature representation x t , to obtain the gradient matrix δG t :

[0042]

[0043] where, represents the partial derivative.

[0044] As a preferred embodiment of the first aspect, in step S3, the specific form of the Jocobian matrix is:

[0045]

[0046] where f1,…,f mt respectively represent the 1,…,m t th classification function; x t1 ,…,xtn respectively represent the first, …, tnth interaction feature representation; tn represents the number of interaction feature representations; m t represents the number of cognitive validity classifications; represents the Jocobian matrix.

[0047] As a preferred embodiment of the above first aspect, in step S3, the function form of the priori knowledge regular term R is:

[0048]

[0049] wherein S represents the number of multi-modal interaction fusion hypergraph representations; represents the Jocobian matrix of the classification function with respect to the sth interaction feature representation; K represents a binary matrix of priori knowledge credibility, wherein an element of 1 represents high credibility and an element of 0 represents low credibility; represents the Hadamard product operation. δG s represents the gradient matrix of the sth interaction feature representation related priori knowledge; ∈ represents the acceptable error;

[0050] As a preferred embodiment of the above first aspect, in step S3, the total loss function L total has the form:

[0051] L total = L CE + γR

[0052] wherein L CE represents the cross-entropy loss function, and γ is the weight of the regular term.

[0053] In a second aspect, the present application provides a data and knowledge collaborative driven cognitive validity calculation model enhancement system, which comprises:

[0054] a hypergraph representation acquisition module, configured to acquire behavior data and brain neural data of students in a teaching interpersonal interaction scenario, extract key features from the two kinds of data, and construct a behavior correlation matrix and a brain neural correlation matrix respectively, which correspond to a behavior hypergraph representation and a brain neural hypergraph representation;

[0055] a hypergraph representation fusion module, configured to perform Fourier transform on a Gaussian kernel function through a multi-core learning method, calculate a mapping function explicitly, and calculate the behavior hypergraph representation and the brain neural hypergraph representation with the mapping function respectively, so as to map the behavior hypergraph representation and the brain neural hypergraph representation to a common high-dimensional feature space to obtain an initial interaction fusion hypergraph representation, and use a mixed L 21 norm constraint optimization on the kernel weight of each Gaussian kernel function. 21The norm-constrained Gaussian kernel function is obtained by weighted combination, an optimized weighted Gaussian kernel function is obtained, Fourier transform is carried out on the optimized weighted Gaussian kernel function, a new mapping function is obtained, and the initial interactive fusion hypergraph representation and the new mapping function are calculated to obtain a multi-modal interactive fusion hypergraph representation.

[0056] The model obtaining module is configured to train the constructed cognitive validity calculation model, obtain an interactive feature representation by performing a hypergraph attention convolution operation on the multi-modal interactive fusion hypergraph representation in the training process of the cognitive validity calculation model, and predict cognitive achievements in teaching interpersonal interaction by performing a classification function on the interactive feature representation.

[0057] The cognitive validity classification module is configured to deploy the trained cognitive validity calculation model to a teaching application platform supporting interpersonal interaction, obtain interactive data between teachers and students in real time, input the interactive data between the teachers and the students into the trained cognitive validity calculation model to evaluate the cognitive validity of teaching interaction, and generate targeted teaching optimization and learning intervention strategies according to the cognitive validity result output by the cognitive validity calculation model, so as to provide auxiliary support for learning subjects in teaching and learning.

[0058] Compared with the prior art, the present application has the following beneficial effects:

[0059] 1) Prior knowledge integration: The present application integrates the prior knowledge related to the cognitive field into the cognitive validity calculation model in a regularized manner, innovatively introduces domain knowledge, and thus promotes the mapping relationship learned by the cognitive validity calculation model to be more consistent with the cognitive rules in the field. 2) Gradient matrix and Jacobian matrix utilization: By utilizing the gradient matrix of prior knowledge and the Jacobian matrix of the cognitive validity calculation model, the present application successfully obtains the prior knowledge regularization term. This step not only provides guidance for the training process of the cognitive validity calculation model, but also provides a more accurate method for introducing domain knowledge into the cognitive validity calculation model. 3) Cognitive validity calculation model training guidance enhancement: By calculating the prior knowledge regularization term and adding it to the loss function of the cognitive validity calculation model, the present application introduces the guidance of domain knowledge in the training process of the cognitive validity calculation model, thereby enhancing the guidance and accuracy of the cognitive validity calculation model in learning the mapping relationship. 4) Data efficient processing and generalization: The cognitive validity calculation model constructed by the present application has higher data processing efficiency and generalization ability. Through the integration of prior knowledge, the cognitive validity calculation model is more likely to adapt to different data sources and situations, improving its flexibility in practical applications.

[0060] Compared with traditional single-modal learning methods, the present application has the following advantages: based on the gradient matrix of prior knowledge and the Jacobian matrix of the model, the prior knowledge regularization term is obtained to guide and constrain the training process of the cognitive validity calculation model, and a deep interactive cognitive validity calculation model with efficient data processing and generalization is constructed. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 The step flowchart of the method of the present application is shown in the figure.

[0062] Figure 2 The module schematic diagram of the system of the present application is shown in the figure. DETAILED DESCRIPTION

[0063] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. In the following description, a large number of specific details are set forth in order to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the spirit of the present application, so the present application is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present application can be combined accordingly without conflict.

[0064] In the description of the present application, it should be understood that the terms "first", "second" are only used for differentiation purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can be explicitly or implicitly included at least one of the features.

[0065] To solve the problems of poor generalization and data dependence of existing data-driven cognitive validity calculation models, in a preferred implementation manner of the present application, a data and knowledge collaborative driven cognitive validity calculation model enhancement method is provided, as shown in Figure 1 The method comprises steps S1-S4. The specific implementation manners of steps S1-S4 are described in detail below.

[0066] S1, in a teaching interpersonal interaction scenario, obtaining behavior data and brain neural data of students, extracting key features from the two kinds of data, respectively constructing behavior correlation matrix and brain neural correlation matrix, corresponding to behavior hypergraph representation and brain neural hypergraph representation.

[0067] It should be noted that the specific implementation process of step S1 of the present application comprises:

[0068] S11, using a video recording system and motion capture technology to obtain behavior data of students, the behavior data containing body movements and expressions of students; using computer vision algorithm (such as OpenPose) to extract key point positions of human body movements and expression features from the behavior data obtained from the video recording system, and extracting dynamic features (such as action duration and action repetition frequency) from the behavior data obtained from the motion capture technology; taking the key point positions of human body movements and expression features and dynamic features as the key features of the behavior data.

[0069] S12, using electroencephalogram (EEG) and functional magnetic resonance imaging (fMRI) technology to obtain brain neural data of students, extracting features from the brain neural data of students by performing frequency analysis (such as Fourier transform) and correlation analysis, obtaining key features of the brain neural data, the key features of the brain neural data including frequency, amplitude and neural connectivity features of the brain neural data.

[0070] S13, taking the key features of the behavior data extracted in step S11 as the vertices in the behavior hypergraph, constructing the hyperedges in the behavior hypergraph according to the correlation between the behavior data, taking the key features of the brain neural data extracted in step S12 as the vertices in the brain neural hypergraph, and constructing the hyperedges in the brain neural hypergraph according to the correlation between the brain neural data.

[0071] It should be noted that in step S13 of the embodiment, the correlation between the behavior data and the correlation between the brain nerve data refer to time series analysis and evaluation of the functional similarity, spatial proximity and dynamic relationship between features, thereby constructing the behavior hypergraph and the brain nerve hypergraph.

[0072] S14, generating a behavior correlation matrix to depict the relationship between the vertices and hyperedges in the behavior hypergraph, taking the behavior correlation matrix as the behavior hypergraph representation, and constructing a brain nerve correlation matrix to depict the relationship between the vertices and hyperedges in the brain nerve hypergraph, taking the brain nerve correlation matrix as the brain nerve hypergraph representation.

[0073] It should be noted that in step S14 of the embodiment, the behavior hypergraph representation formed by the matrix can accurately capture the complex interaction between behavior features. Such brain nerve hypergraph representation helps to deeply analyze and recognize patterns of neural data. In addition to constructing the respective correlation matrices for the two hypergraphs in the above steps, a behavior adjacency matrix can also be constructed for the behavior hypergraph, and the elements in the behavior adjacency matrix reflect the connection strength between the vertices in the behavior hypergraph, and a brain nerve adjacency matrix can be created for the brain nerve hypergraph to express the interaction between the vertices of the neural data.

[0074] S2, using a multi-kernel learning method, performing Fourier transform on a Gaussian kernel function for explicitly calculating a mapping function, and calculating the behavior hypergraph representation and the brain nerve hypergraph representation with the mapping function respectively, so as to map the behavior hypergraph representation and the brain nerve hypergraph representation to a common high-dimensional feature space, obtaining an initial interactive fusion hypergraph representation, and using mixed L 21 norm constraint optimization on the kernel weight of each Gaussian kernel function to obtain an optimized weighted Gaussian kernel function. 21 The optimized weighted Gaussian kernel function is obtained by weighted combination of the Gaussian kernel function after mixed L

[0075] It should be noted that the specific implementation process of step S2 of the embodiment includes:

[0076] S21, selecting a Gaussian kernel function as a basic kernel function to process the nonlinear characteristics of the behavior hypergraph representation and the brain nerve hypergraph representation; for two data points from the same hypergraph representation, the Gaussian kernel function is defined as:

[0077]

[0078] wherein, μ is a kernel width parameter, used to control the smoothness of the Gaussian kernel function; K(x, y) represents the Gaussian kernel function of two data points from the same hypergraph representation; x and y represent two data points from the same hypergraph representation;

[0079] S22, applying a Fourier transform on each Gaussian kernel function, explicitly calculating a mapping function, and calculating the behavior hypergraph representation and the brain neural hypergraph representation with the mapping function respectively, so as to map the behavior hypergraph representation and the brain neural hypergraph representation to a common high-dimensional feature space, and obtain an initial interactive fusion hypergraph representation;

[0080] For the Gaussian kernel function K(x, ·), the mapping function is defined as:

[0081]

[0082] wherein, K(x, ·) represents the Gaussian kernel function of two data points from different hypergraph representations; FFT(·) represents a Fourier transform, used to convert the Gaussian kernel function from a time domain to a frequency domain, explicitly calculate the position of each data point in the high-dimensional space, and thus convert the local feature of each data point into a global feature;

[0083] S23, using a mixed L 21 norm constraint to optimize the weight of each Gaussian kernel function, so as to make full use of the complementary information between different Gaussian kernel functions; wherein, the mixed L 21 norm constraint is defined as:

[0084]

[0085] wherein, W i represents the weight matrix of the i-th Gaussian kernel function, A is a design matrix, B is a target matrix, λ is a regularization parameter, used to prevent overfitting; W represents the weight matrix of all Gaussian kernel functions; represents the square of the L2 norm.

[0086] S24, applying a mixed L 21 norm constraint to the Gaussian kernel function, and obtaining an optimized weighted Gaussian kernel function K combined :

[0087]

[0088] wherein, w i represents the weight of the i-th Gaussian kernel function after applying the mixed L 21 norm constraint, used to represent the importance of different Gaussian kernel functions in the fusion feature; K i(x,y) represents the weighted Gaussian kernel function after applying the mixed L21 the i-th Gaussian kernel function after norm constraint; n represents the number of Gaussian kernel functions.

[0089] In step S2, in order to maintain the uniqueness of the behavior modal and the brain neural modal, the weight of different Gaussian kernel functions is assigned by using a soft-max function. Therefore, a mixed L 21 The weight of the i-th Gaussian kernel function after norm constraint is specifically:

[0090]

[0091] wherein β represents a learnable parameter; d i , d j respectively represent the discriminability indexes of the i-th and j-th Gaussian kernel functions.

[0092] S25, performing Fourier transform on the optimized weighted Gaussian kernel function to obtain a new mapping function, and calculating the initial interactive fusion hypergraph representation and the new mapping function to obtain a multi-modal interactive fusion hypergraph representation.

[0093] S3, training the constructed cognitive effectiveness calculation model, in the training process of the cognitive effectiveness calculation model, the multi-modal interactive fusion hypergraph representation is subjected to a hypergraph attention convolution operation of the cognitive effectiveness calculation model, effective extraction of the multi-modal interactive fusion hypergraph representation is realized, an interactive feature representation is obtained, the cognitive achievement in the teaching interpersonal interaction is predicted by the classification function f of the cognitive effectiveness calculation model on the interactive feature representation, the partial derivative of the classification function on the interactive feature representation is solved, a Jocobian matrix is obtained, the related knowledge in the cognitive field is taken as a priori to construct a gradient matrix of priori knowledge, so as to enhance the accuracy of the cognitive effectiveness calculation model in predicting the cognitive effectiveness, the gradient matrix of priori knowledge and the Jocobian matrix are used to construct a priori knowledge regularization term, a cross-entropy loss function and the weighted priori knowledge regularization term are added as a total loss function of the cognitive effectiveness calculation model, the parameters of the cognitive effectiveness calculation model are updated based on minimizing the total loss function, and the iterative training is continuously performed until the total loss function converges, and finally the trained cognitive effectiveness calculation model is obtained.

[0094] In this step, in view of the characteristics that there is mutual influence between the teaching interpersonal interaction modes, the correlation degree between the interaction modes is quantified through an attention mechanism. Specifically, the correlation degree of different interaction modes in the teaching interpersonal interaction scene can be quantified by using attention scores. These attention scores are calculated through a parameterized attention function a(·), based on the feature vectors and the adjacency relationship of the interactive hypergraph representation, the mutual influence between the nodes in the interactive hypergraph representation is evaluated, and the calculation method of the attention score is:

[0095]

[0096] wherein e pq is the attention score between nodes p and q, respectively represent the weight matrix of the current layer, x p and x q are the feature vectors of the corresponding nodes.

[0097] It should be noted that in step S3, the hypergraph attention convolution of the lth layer can be expressed as:

[0098]

[0099] wherein, and respectively represent the input feature and the output feature of the lth layer hypergraph attention convolution layer; σ is a nonlinear activation function; and respectively represent the degree matrix of the hypergraph representation of the interactive fusion of multiple modalities; H represents the association matrix of the hypergraph representation of the interactive fusion of multiple modalities; W (l) represents the weight matrix of the lth layer hypergraph attention convolution layer; P (l) is the parameter matrix of the lth layer hypergraph attention convolution layer; N represents the number of vertices of the hypergraph representation of the interactive fusion of multiple modalities; M represents the number of hyperedges of the hypergraph representation of the interactive fusion of multiple modalities; F represents the cognitive effectiveness calculation model; F (l) represents the feature dimension of the lth layer of the cognitive effectiveness calculation model, F (l+1) the feature dimension of the (l+1)th layer of the cognitive effectiveness calculation model; T represents matrix transposition; -1 represents the inverse matrix.

[0100] In this embodiment, the feature representation of the nodes is updated layer by layer using the hypergraph attention convolution layer defined above, so as to obtain and deepen the understanding of the correlation degree between the interpersonal interaction patterns, and realize effective extraction of the interactive feature representation of multiple modalities.

[0101] It should be noted that in step S3, the specific way of constructing the gradient matrix of the prior knowledge is:

[0102] S31, taking the knowledge related to the cognitive field as the prior knowledge, using the utility function U(x t ) to represent the high and low of the cognitive effect, for converting the form of the prior knowledge into mathematical expression.

[0103] In this embodiment, on the basis of predicting the cognitive effectiveness of the interactive feature representation, the knowledge related to the cognitive field (such as the conclusion obtained by cognitive psychology) is selected as the prior knowledge. The utility function U(x t ) is introduced, for example, in the teaching interpersonal interaction, it is known that group discussion can enhance the critical thinking of students, various quantitative indicators of group discussion are introduced, such as discussion frequency x t1and student engagement x t2 , then U(x t ) = a1x t1 + a2x t2 , where x t represents the interaction feature representation, and a1 and a2 are weights pre-set based on prior knowledge.

[0104] S32, compute the partial derivative of the utility function U(x t ) with respect to the interaction feature representation x t , to obtain a gradient matrix δG t :

[0105]

[0106] where denotes the partial derivative.

[0107] In this embodiment, when the utility function U(x t ) = a1x t1 + a2x t2 , the specific form of the gradient matrix is:

[0108]

[0109] The gradient matrix δG t directly reflects the sensitivity of each interaction feature representation to the cognitive effect, where the gradient value of each component guides the cognitive validity calculation model to prioritize features more closely related to the cognitive effect in the update process.

[0110] It should be noted that in step S3, the specific way to obtain the Jacobian matrix is:

[0111] The cognitive validity calculation model is used to analyze and predict cognitive outcomes in teaching interpersonal interactions. The classification function f receives the interaction feature representation x t in S4 and takes it as input, and outputs one or more prediction results, which represent various aspects of cognitive effects, such as verbal communication, body movements, facial expressions, attention, and imitation, etc. Specifically, the partial derivative of the classification function f with respect to the interaction feature representation x t is solved to obtain the Jacobian matrix Each element in the Jacobian matrix is the partial derivative of the classification function f with respect to each element in the input interaction feature representation x t . For the classification function The Jacobian matrix is defined as:

[0112]

[0113] wherein, respectively represent the 1st, …, m t classification functions; x t1 ,…, x tn respectively represent the 1st, …, tninteraction feature representations; tnrepresents the number of interaction feature representations; m t represents the number of cognitive validity classifications.

[0114] It should be noted that the priori knowledge regular term is calculated in step S3 of the present application and added to the loss function of the cognitive validity calculation model to guide the training process of the cognitive validity calculation model.

[0115] After constructing the Jacobian matrix of the cognitive validity calculation model with respect to the s-th input, the dot product of the binary matrix K of the credibility and the gradient matrix of the priori knowledge is subtracted, and then the L1 norm is taken, so as to obtain the priori knowledge regular term. The function form of the priori knowledge regular term R is:

[0116]

[0117] wherein, S represents the number of multi-modal interaction fusion hypergraph representations; represents the Jacobian matrix of the classification function with respect to the s-th interaction feature representation; K represents the binary matrix of the credibility of the priori knowledge, wherein the element of 1 represents high trust and the element of 0 represents low trust; ⊙ represents the Hadamard product operation; δG s represents the gradient matrix of the s-th interaction feature representation related to the priori knowledge; ∈ represents the acceptable error.

[0118] The above priori knowledge regular term is used to guide and constrain the training process of the cognitive validity calculation model, so as to improve the performance and generalization of the cognitive validity calculation model. The total loss function L total of the final cognitive validity calculation model is:

[0119] L total = L CE + γR

[0120] wherein, L CE represents the cross-entropy loss function, and γ is the weight of the regular term, which is used to balance the influence of the basic cross-entropy loss function and the regular term.

[0121] S4, deploying the trained cognitive effectiveness calculation model into a teaching application platform supporting interpersonal interaction, the teaching application platform acquiring interaction data between teachers and students in real time, inputting the interaction data between teachers and students into the trained cognitive effectiveness calculation model, analyzing the multi-modal interaction data by the trained cognitive effectiveness calculation model, and evaluating the cognitive effectiveness of the current teaching interaction in real time. According to the cognitive effectiveness result output by the cognitive effectiveness calculation model, the teaching application generates targeted teaching optimization and learning intervention strategies to provide auxiliary support for learning subjects and promote the efficiency of the teaching process. The interaction data between teachers and students are the behavior data and brain neural data of students and the behavior data and brain neural data of teachers.

[0122] In step S4 of the present application, the cognitive effectiveness of teaching interaction refers to the result reflecting the cognitive level of students' learning effect during the teaching process of teachers through students' speech communication, body movements, facial expressions after class and attention, etc. In addition, the teaching application platform will continuously acquire the interaction data and the evaluation result of the cognitive effectiveness calculation model during the teaching process, and further train and optimize the cognitive effectiveness calculation model by using these data to realize the iterative optimization of the cognitive effectiveness calculation model.

[0123] In addition, it should be noted that the data and knowledge collaborative driving cognitive effectiveness calculation model enhancement method in the above embodiment can be essentially executed by a computer program or module. Therefore, based on the same inventive concept, another preferred embodiment of the present application also provides a data and knowledge collaborative driving cognitive effectiveness calculation model enhancement system corresponding to the data and knowledge collaborative driving cognitive effectiveness calculation model enhancement method provided by the above embodiment, as shown in the following figure, which includes four basic modules, respectively: Figure 2

[0124] 1) Hypergraph representation acquisition module, used for acquiring the behavior data and brain neural data of students in the teaching interpersonal interaction scene, extracting key features from the two kinds of data, respectively constructing a behavior association matrix and a brain neural association matrix, and corresponding to a behavior hypergraph representation and a brain neural hypergraph representation.

[0125] In the hypergraph representation acquisition module of the present embodiment, multi-modal interaction data in the teaching scene are acquired and integrated, including hypergraph representations of two modalities of behavior level and brain neural level, to provide comprehensive and rich input information for subsequent processing.

[0126] ​2) supergraph representation fusion module, for performing Fourier transform on Gaussian kernel function by multi-kernel learning method, for explicitly calculating mapping function, and calculating behavior supergraph representation and brain neural supergraph representation with mapping function respectively, thereby mapping behavior supergraph representation and brain neural supergraph representation to a common high-dimensional feature space to obtain initial interactive fusion supergraph representation, using mixed L 21 norm constraint optimization, the mixed L 21 norm constraint is applied to the Gaussian kernel function, and the optimized weighted Gaussian kernel function is obtained by weighted combination, and Fourier transform is performed on the optimized weighted Gaussian kernel function to obtain a new mapping function, and the initial interactive fusion supergraph representation is calculated with the new mapping function to obtain a multi-modal interactive fusion supergraph representation.

[0127] In the supergraph representation fusion module of the embodiment, multi-kernel complementary information is integrated for feature fusion to obtain a multi-modal interactive fusion supergraph representation, so as to improve the dimension and richness of feature representation.

[0128] 3) model acquisition module, for training the constructed cognitive effectiveness calculation model, in the training process of the cognitive effectiveness calculation model, the multi-modal interactive fusion supergraph representation is subjected to supergraph attention convolution operation of the cognitive effectiveness calculation model to obtain interactive feature representation, the interactive feature representation is subjected to classification function of the cognitive effectiveness calculation model to predict cognitive achievement in teaching interpersonal interaction; the partial derivative of the classification function to the interactive feature representation is solved to obtain Jocobian matrix, the related knowledge in the cognitive field is taken as priori to construct gradient matrix of priori knowledge, the gradient matrix of priori knowledge and the Jocobian matrix are used to construct priori knowledge regularization term, the cross-entropy loss function and the weighted priori knowledge regularization term are added as total loss function of the cognitive effectiveness calculation model, the cognitive effectiveness calculation model parameters are updated based on minimizing the total loss function, and the iterative training is continuously performed until the total loss function converges, and finally the trained cognitive effectiveness calculation model is obtained.

[0129] In the model acquisition module of the embodiment, the supergraph attention convolution operation is used to effectively extract the multi-modal interactive fusion supergraph representation; the module uses the strong correlation between interactions to strengthen the weight of key information to obtain interactive feature representation, so as to more accurately capture interactive mode and related information; the domain knowledge is used as priori to construct gradient matrix of related priori knowledge, and the priori knowledge regularization term is calculated and introduced into the loss function of the cognitive effectiveness calculation model to constrain the training process of the cognitive effectiveness calculation model, so that it is more consistent with the domain knowledge.

[0130] 4) a cognitive validity classification module for deploying the trained cognitive validity calculation model into a teaching application platform supporting interpersonal interaction, obtaining interaction data between teachers and students in real time, inputting the interaction data between teachers and students into the trained cognitive validity calculation model to evaluate the cognitive validity of teaching interaction, and according to the cognitive validity result output by the cognitive validity calculation model, the teaching application generates targeted teaching optimization and learning intervention strategies to provide auxiliary support for learning subjects; wherein the interaction data between teachers and students is the behavior data and brain neural data of students and the behavior data and brain neural data of teachers.

[0131] In the cognitive validity classification module of the present embodiment, based on the above processing, the module is used for classification by using the trained cognitive validity classification module to distinguish the cognitive validity of teaching interpersonal interaction process elements such as verbal communication, body movement, facial expression, attention and imitation, and further to realize effective classification and evaluation of the cognitive validity of teaching interpersonal interaction.

[0132] Through the organic combination of the above modules, the system can efficiently extract and combine multi-modal information under the collaborative driving of data and knowledge to enhance the performance and interpretability of the cognitive validity calculation model in the teaching scenario.

[0133] The above-described embodiments are only a preferred scheme of the present application, and are not intended to limit the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, any technical solutions obtained by equivalent replacement or equivalent transformation shall fall within the protection scope of the present application.

Claims

1. A method for enhancing a cognitive validity calculation model driven by data and knowledge collaboration, characterized in that: The following steps are involved: S1. In the context of interpersonal interaction in teaching, obtain students' behavioral data and brain neural data, extract key features from the two types of data, and construct behavioral association matrices and brain neural association matrices respectively, which serve as behavioral hypergraph representations and brain neural hypergraph representations respectively; S2. Using the multi-kernel learning method, Fourier transform is performed on the Gaussian kernel function to explicitly calculate the mapping function. The behavioral hypergraph representation and the brain neural hypergraph representation are calculated with the mapping function respectively, thereby mapping the behavioral hypergraph representation and the brain neural hypergraph representation to a common high-dimensional feature space to obtain the initial interactive fusion hypergraph representation. The kernel weight of each Gaussian kernel function is calculated using the hybrid L 21 Norm-constrained optimization, applying mixed L 21 The norm-constrained Gaussian kernel function is weightedly combined to obtain an optimized weighted Gaussian kernel function. The optimized weighted Gaussian kernel function is Fourier transformed to obtain a new mapping function. The initial interactive fusion hypergraph representation and the new mapping function are calculated to obtain a multimodal interactive fusion hypergraph representation. S3. Train the constructed cognitive validity calculation model. During the training process of the cognitive validity calculation model, the multimodal interaction fusion hypergraph representation is subjected to the hypergraph attention convolution operation of the cognitive validity calculation model to obtain the interaction feature representation. The interaction feature representation is subjected to the classification function of the cognitive validity calculation model to predict the cognitive outcomes in the teaching interpersonal interaction. The partial derivative of the classification function with respect to the interaction feature representation is solved to obtain the Jocobian matrix. The relevant knowledge in the cognitive field is used as a priori to construct the gradient matrix of the priori knowledge. The gradient matrix of the priori knowledge and the Jocobian matrix are used to construct the priori knowledge regularization term. The cross entropy loss function and the weighted priori knowledge regularization term are added as the total loss function of the cognitive validity calculation model. The parameters of the cognitive validity calculation model are updated based on minimizing the total loss function. The training is continuously iterated until the total loss function converges, and finally a trained cognitive validity calculation model is obtained. S4. Deploy the trained cognitive validity calculation model to a teaching application platform that supports interpersonal interaction, obtain the interaction data between teachers and students in real time, input the interaction data between teachers and students into the trained cognitive validity calculation model to evaluate the cognitive validity of the teaching interaction, and generate targeted teaching optimization and learning intervention strategies based on the cognitive validity results output by the cognitive validity calculation model to provide auxiliary support for teaching and learning for learning subjects; among them, the interaction data between teachers and students include students' behavioral data and brain neural data as well as teachers' behavioral data and brain neural data.

2. The data and knowledge collaboratively driven cognitive validity calculation model enhancement method according to claim 1, characterized in that: The specific implementation process of step S1 includes: S11. Obtaining student behavioral data using a video recording system and motion capture technology, wherein the behavioral data includes the student's body movements and facial expressions; extracting key points of human body movements and facial expression features from the behavioral data obtained by the video recording system using a computer vision algorithm, and extracting dynamic features from the behavioral data obtained by the motion capture technology; and using the key points of human body movements, facial expression features, and dynamic features as key features of the behavioral data; S12. Acquire the student's cranial nerve data using electroencephalogram (EEG) and functional magnetic resonance imaging (FMRI) techniques, extract features from the student's cranial nerve data by performing frequency analysis and correlation analysis, and obtain key features of the cranial nerve data, wherein the key features of the cranial nerve data include frequency, amplitude, and neural connectivity features of the cranial nerve data; S13, using the key features of the behavioral data extracted in step S11 as vertices in the behavioral hypergraph, constructing hyperedges in the behavioral hypergraph based on the correlation between the behavioral data, and using the key features of the brain nerve data extracted in step S12 as vertices in the brain nerve hypergraph, constructing hyperedges in the brain nerve hypergraph based on the correlation between the brain nerve data; S14. Generate a behavioral association matrix to depict the relationship between vertices and hyperedges in the behavioral hypergraph, use the behavioral association matrix as the behavioral hypergraph representation, construct a brain neural association matrix to depict the relationship between vertices and hyperedges in the brain neural hypergraph, and use the brain neural association matrix as the brain neural hypergraph representation.

3. The data and knowledge collaboratively driven cognitive validity calculation model enhancement method according to claim 1, characterized in that: The specific implementation process of step S2 includes: S21. Select the Gaussian kernel function as the basic kernel function to handle the nonlinear characteristics of behavioral hypergraph representation and brain neural hypergraph representation. For two data points from the same hypergraph representation, the Gaussian kernel function is defined as: Where μ is the kernel width parameter, which is used to control the smoothness of the Gaussian kernel function; K(x,y) represents the Gaussian kernel function of two data points from the same hypergraph representation; x and y represent two data points from the same hypergraph representation; S22. Apply Fourier transform to each Gaussian kernel function, explicitly calculate the mapping function, and calculate the behavioral hypergraph representation and the brain neural hypergraph representation with the mapping function respectively, thereby mapping the behavioral hypergraph representation and the brain neural hypergraph representation to a common high-dimensional feature space to obtain an initial interactive fusion hypergraph representation; For the Gaussian kernel function K(x,·), the mapping function Defined as: where K(x,·) represents the Gaussian kernel function of two data points from different hypergraph representations; FFT(·) represents Fourier transform; S23, use mixed L for the kernel weight of each Gaussian kernel function 21 Norm constraint optimization of the weights of each Gaussian kernel function, mixed L 21 The norm constraint is defined as: Among them, W i represents the weight matrix of the i-th Gaussian kernel function, A is the design matrix, B is the target matrix, and λ is the regularization parameter; W represents the weight matrix of all Gaussian kernel functions; represents the square of L2 norm; S24, apply mixed L 21 The Gaussian kernel function after norm constraint is weighted combination to obtain the optimized weighted Gaussian kernel function K combined : Among them, w i Indicates the application of mixed L 21 The weight of the i-th Gaussian kernel function after norm constraint; K i(x,y) Indicates the application of mixed L 21 The i-th Gaussian kernel function after norm constraint; n represents the number of Gaussian kernel functions; S25. Perform Fourier transform on the optimized weighted Gaussian kernel function to obtain a new mapping function, calculate the initial interactive fusion hypergraph representation and the new mapping function to obtain a multimodal interactive fusion hypergraph representation.

4. The data and knowledge collaboratively driven cognitive validity calculation model enhancement method according to claim 3, characterized in that: In step S2, the mixed L 21 After the norm constraint, the weight of the i-th Gaussian kernel function is specifically: Among them, β represents the learnable parameter; d i ,d j Represent the discrimination index of the i-th and j-th Gaussian kernel functions respectively.

5. The data and knowledge collaboratively driven cognitive validity calculation model enhancement method according to claim 1, characterized in that: In step S3, the hypergraph attention convolution of layer l can be expressed as: Among them, X (l) and X (l+1) Denote the input features and output features of the lth layer of hypergraph attention convolutional layer respectively; σ is a nonlinear activation function; D and B denote the degree matrices of the multimodal interactive fusion hypergraph representation vertices and hyperedges respectively; H denotes the association matrix of the multimodal interactive fusion hypergraph representation hyperedges to vertices; W (l) represents the weight matrix of the lth hypergraph attention convolution layer; P (l) is the parameter matrix of the lth hypergraph attention convolutional layer; T represents the matrix transpose; -1 represents the inverse matrix.

6. The data and knowledge collaboratively driven cognitive validity calculation model enhancement method according to claim 1, characterized in that: In step S3, the specific method of constructing the gradient matrix of prior knowledge is: S31, taking the knowledge related to the cognitive domain as prior knowledge, using the utility function U(x t ) indicates the level of cognitive effect and is used to transform prior knowledge into mathematical expression; S32. Calculate the utility function U(x t ) About the interactive feature representation x t The partial derivative of the gradient matrix δG is obtained t : in, It means to find partial derivatives.

7. The data and knowledge collaboratively driven cognitive validity calculation model enhancement method according to claim 6, characterized in that: In step S3, the specific form of the Jocobian matrix is: in, Represent the 1st,…,mth t classification function; x t1 ,…,x tn Respectively represent the 1st,…,tnth interactive feature representations; tn represents the number of interactive feature representations; m t represents the number of cognitive validity categories; Represents the Jacobian matrix.

8. The data and knowledge collaboratively driven cognitive validity calculation model enhancement method according to claim 7, characterized in that: In step S3, the function form of the prior knowledge regularization term R is: Where S represents the number of multimodal interactive fusion hypergraph representations; represents the Jocobian matrix of the classification function with respect to the sth interaction feature representation; K represents the binary matrix of the prior knowledge credibility, in which an element of 1 represents high trust and an element of 0 represents low trust; ⊙ represents the Hadamard product operation, δG s represents the gradient matrix of the prior knowledge about the sth interaction feature representation; ∈ represents the acceptable error.

9. The data and knowledge collaboratively driven cognitive validity calculation model enhancement method according to claim 8, characterized in that: In step S3, the total loss function L total The form is: IT total =L CE +γR Among them, L CE represents the cross entropy loss function, and γ is the weight of the regularization term.

10. A cognitive validity calculation model enhancement system driven by data and knowledge collaboration, characterized by: include: The hypergraph representation acquisition module is used to obtain students' behavioral data and brain neural data in the context of interpersonal interaction in teaching, extract key features from the two types of data, and construct behavioral association matrices and brain neural association matrices respectively, which serve as behavioral hypergraph representations and brain neural hypergraph representations respectively; The hypergraph representation fusion module is used to perform Fourier transform on the Gaussian kernel function through the multi-kernel learning method, and is used to explicitly calculate the mapping function. The behavioral hypergraph representation and the brain neural hypergraph representation are respectively calculated with the mapping function, thereby mapping the behavioral hypergraph representation and the brain neural hypergraph representation to a common high-dimensional feature space to obtain the initial interactive fusion hypergraph representation. The kernel weight of each Gaussian kernel function is used to use the mixed L 21 Norm-constrained optimization, applying mixed L 21 The norm-constrained Gaussian kernel function is weightedly combined to obtain an optimized weighted Gaussian kernel function. The optimized weighted Gaussian kernel function is Fourier transformed to obtain a new mapping function. The initial interactive fusion hypergraph representation and the new mapping function are calculated to obtain a multimodal interactive fusion hypergraph representation. The model acquisition module is used to train the constructed cognitive validity calculation model. During the training process of the cognitive validity calculation model, the multimodal interaction fusion hypergraph representation is subjected to the hypergraph attention convolution operation of the cognitive validity calculation model to obtain the interaction feature representation. The interaction feature representation is subjected to the classification function of the cognitive validity calculation model to predict the cognitive outcomes in the teaching interpersonal interaction. The partial derivative of the classification function with respect to the interaction feature representation is solved to obtain the Jocobian matrix. The relevant knowledge in the cognitive field is used as the prior to construct the gradient matrix of the prior knowledge. The gradient matrix of the prior knowledge and the Jocobian matrix are used to construct the prior knowledge regularization term. The cross entropy loss function and the weighted prior knowledge regularization term are added as the total loss function of the cognitive validity calculation model. The parameters of the cognitive validity calculation model are updated based on minimizing the total loss function. The training is iterated continuously until the total loss function converges, and finally a trained cognitive validity calculation model is obtained. The cognitive validity classification module is used to deploy the trained cognitive validity calculation model to a teaching application platform that supports interpersonal interaction, obtain the interaction data between teachers and students in real time, input the interaction data between teachers and students into the trained cognitive validity calculation model to evaluate the cognitive validity of the teaching interaction, and generate targeted teaching optimization and learning intervention strategies based on the cognitive validity results output by the cognitive validity calculation model to provide auxiliary support for teaching and learning for learning subjects; among them, the interaction data between teachers and students are the students' behavioral data and brain neural data as well as the teachers' behavioral data and brain neural data.

Images