Learning state detection method based on cross-modal representation learning

Abstract

The embodiment of the invention discloses a learning state detection method based on cross-modal representation learning, and relates to the technical field of teaching informationization, and the method comprises the steps: obtaining first video information collected by an RGB camera and second video information collected by a dynamic visual camera, carrying out the processing of the first video information and the second video information, and obtaining a first image sequence and a second image sequence, and inputting the first image sequence and the second image sequence into a trained learning state detection network model, performing identification to obtain a facial expression identification result and a learning behavior identification result of each learner in the target learning group, and determining a learning state detection result of each learner in the target learning group at a corresponding moment. By adopting the method provided by the invention, the overall learning of the learning group is estimated on the basis of the facial emotion recognition result and the learning behavior recognition result of each learner, the collective learning effect of the learners can be accurately evaluated, and a teacher can know the real-time learning state of each learning group in real time.

Description

Technical Field

[0001] The present application relates to the field of teaching information technology, and in particular to a learning state detection method based on cross-modal representation learning. Background Art

[0002] Students can improve their learning efficiency by forming study groups or teams to study together, sharing goals, motivating each other, and adjusting learning strategies together. With the development of educational technology, the traditional classroom teaching model has gradually shifted to an intelligent and personalized learning environment. In this context, it is difficult to evaluate the learning status of each student in the study group or team, and it is difficult to determine whether each student is engaged in learning, which faces many challenges.

[0003] Traditional learning status assessment methods rely on the subjective judgment of teachers and learners, and there are problems such as too strong subjectivity and untimely assessment. In addition, unlike the traditional learning environment, in the learning process of learning groups and learning teams, the learning unit is the corresponding learning group rather than a single student. Therefore, the change of the assessment subject brings greater challenges to the perception of learning status in team learning.

[0004] With the in-depth integration of information technology and education and teaching, the technology-supported smart classroom plays an important role in optimizing learning outcomes. Compared with traditional classrooms, smart classrooms provide convenience for collecting learners' learning behaviors. However, in the process of team learning, there are still problems such as inaccurate and untimely perception of the learning status of the learning group.

[0005] Therefore, there is a need for an auxiliary means that is conducive to improving the learning quality in the team learning process and enhancing students' team learning collaboration ability. Summary of the invention

[0006] The embodiment of the present application provides a learning state detection method based on cross-modal representation learning to solve the defects of the above-mentioned related technologies. The technical solution is as follows:

[0007] In a first aspect, an embodiment of the present application provides a learning state detection method based on cross-modal representation learning, comprising:

[0008] Acquire first video information of a target learning group captured by an RGB camera and second video information of the target learning group captured by a dynamic vision camera;

[0009] Processing the first video information to obtain a first image sequence, and processing the second video information to obtain a second image sequence;

[0010] Inputting the first image sequence and the second image sequence into a trained learning state detection network model, obtaining a facial expression recognition result of each learner in the target learning group through the first sub-model of the learning state detection network model based on the first image sequence recognition, and obtaining a learning behavior recognition result of each learner in the target learning group through the second sub-model of the learning state detection network model based on the second image sequence recognition;

[0011] The learning state detection result of each learner in the target learning group at a corresponding moment is determined based on the facial expression recognition result and the learning behavior recognition result.

[0012] In an optional solution of the first aspect, after obtaining the first video information of the target learning group photographed by the RGB camera and the second video information of the target learning group photographed by the dynamic vision camera, it also includes:

[0013] Decomposing the first video information into a first sub-video set consisting of a plurality of sub-video segments in time sequence, and decomposing the second video information into a second sub-video set consisting of a plurality of sub-video segments in time sequence; wherein in the same sub-video set, there is an overlapping segment of a preset time length between every two sub-videos that are adjacent to each other in time sequence;

[0014] Extracting a preset number of video frames from the first sub-video set based on a preset time interval to obtain a first video frame set, and extracting a preset number of video frames from the second sub-video set based on a preset time interval to obtain a second video frame set;

[0015] The step of obtaining a first image sequence based on the first video information processing and obtaining a second image sequence based on the second video information processing includes:

[0016] The first image sequence is obtained based on the first video frame set, and the second image sequence is obtained based on the second video frame set.

[0017] In an optional solution of the first aspect, before obtaining the first image sequence based on the first video information processing and obtaining the second image sequence based on the second video information processing, the method further includes:

[0018] Performing face detection on each video frame in the first video information, cropping the video frames in which faces are recognized, and retaining the facial image areas of all the video frames in which faces are recognized;

[0019] Performing denoising processing on each video frame in the second video information, and outputting the denoised video frame;

[0020] The facial image area and the denoised video frame are respectively cropped to a preset pixel size.

[0021] In an optional solution of the first aspect, obtaining a facial expression recognition result of each learner in the target learning group based on the first image sequence recognition by the first submodel of the learning state detection network model includes:

[0022] Performing feature extraction on each image frame in the first image sequence to obtain facial spatial features corresponding to each image frame;

[0023] The facial spatial features are processed to obtain a static facial spatial feature sequence, and facial spatial features corresponding to adjacent image frames are subtracted and processed to obtain a dynamic facial spatial feature sequence;

[0024] Obtaining a facial change spatial feature sequence based on the static facial spatial feature sequence and the dynamic facial spatial feature sequence;

[0025] Each facial change feature is extracted based on the facial change spatial feature sequence, an expression prediction result tensor is obtained based on the average value of each facial change feature, and an expression category label of the corresponding learner is determined based on the expression prediction result tensor.

[0026] In an optional solution of the first aspect, obtaining a learning behavior recognition result of each learner in the target learning group based on the second image sequence recognition by the second submodel of the learning state detection network model includes:

[0027] Dividing each image frame in the second image sequence into a plurality of image blocks, and adding a position code to each of the image blocks to obtain a visual feature sequence;

[0028] Processing the visual feature sequence sequentially through the frame interaction transformer module and the memory fusion transformer module of the second sub-model to obtain discriminative visual features of the corresponding learning behavior video;

[0029] The cosine similarity between the discriminative visual features of the learning behavior video and each text description feature in a preset learning behavior text description feature sequence is calculated, and the learning behavior category corresponding to the text description feature with the highest cosine similarity is output as the learning behavior recognition result of the corresponding learner.

[0030] In an optional solution of the first aspect, it is characterized in that the determining the learning state detection result of each learner in the target learning group at a corresponding moment based on the facial expression recognition result and the learning behavior recognition result comprises:

[0031] Determine the distribution of facial expression recognition results for each learner, and determine the facial expression parameters of the target learning group based on the facial expression category with the largest proportion in the distribution of facial expression recognition results;

[0032] Determine the distribution of learning behavior identification results for each learner, and determine the learning behavior parameters of the target learning group based on the learning behavior category with the largest proportion in the distribution of learning behavior identification results;

[0033] The learning status detection result of the target learning group at the corresponding moment is obtained by weighted calculation based on the facial expression parameters and the learning behavior parameters.

[0034] In a second aspect, an embodiment of the present application further provides a learning state detection device based on cross-modal representation learning, comprising:

[0035] A data acquisition module, used for acquiring first video information of a target learning group photographed by an RGB camera and second video information of the target learning group photographed by a dynamic vision camera;

[0036] An image processing module, configured to obtain a first image sequence based on the first video information and obtain a second image sequence based on the second video information;

[0037] A detection module, used for inputting the first image sequence and the second image sequence into a trained learning state detection network model, obtaining a facial expression recognition result of each learner in the target learning group through a first sub-model of the learning state detection network model based on the first image sequence recognition, and obtaining a learning behavior recognition result of each learner in the target learning group through a second sub-model of the learning state detection network model based on the second image sequence recognition;

[0038] The detection module is also used to determine the learning state detection result of each learner in the target learning group at the corresponding moment according to the facial expression recognition result and the learning behavior recognition result. In a third aspect, an embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method provided by the first aspect of the embodiment of the present application or any one of the implementations of the first aspect is implemented.

[0039] In a fourth aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method provided by the first aspect of the embodiment of the present application or any one of the implementations of the first aspect.

[0040] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least:

[0041] The embodiment of the present application provides a learning status detection method based on cross-modal representation learning, which estimates the overall learning of a learning group based on the facial emotion recognition results and learning behavior recognition results of each learner, and can accurately evaluate the collective learning effect of learners, which is beneficial for teachers to understand the real-time learning status of each learning group in real time, thereby adjusting the learning method of each learning group or learning team according to the real-time learning status, thereby improving the learning efficiency of team learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 It is a flow chart of a learning state detection method based on cross-modal representation learning according to an embodiment of the present application;

[0044] Figure 2 It is a flow chart of a learning state detection method based on cross-modal representation learning according to an embodiment of the present application;

[0045] Figure 3 It is a flow chart of a learning state detection method based on cross-modal representation learning according to an embodiment of the present application;

[0046] Figure 4 It is a structural schematic diagram of a learning state detection device based on cross-modal representation learning provided in an embodiment of the present application;

[0047] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] The terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or modules is not limited to the listed steps or modules, but may optionally include steps or modules that are not listed, or may optionally include other steps or modules that are inherent to these processes, methods, products or devices.

[0050] It should be noted that the terms "first\second" involved in the present application are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that "first\second" can be interchanged with a specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those described or illustrated herein.

[0051] The present application is described in detail below with reference to specific embodiments.

[0052] Next, combine Figure 1 , introduces the learning state detection method based on cross-modal representation learning provided by the embodiment of the present application. For details, please refer to Figure 1 , Figure 1 A flow chart of a learning state detection method based on cross-modal representation learning provided by an embodiment of the present application is shown, and the method comprises the following steps:

[0053] S101, obtaining first video information of a target learning group captured by an RGB camera and second video information of the target learning group captured by a dynamic vision camera.

[0054] S102: Obtain a first image sequence based on the first video information, and obtain a second image sequence based on the second video information.

[0055] S103, inputting the first image sequence and the second image sequence into a trained learning status detection network model, obtaining facial expression recognition results of each learner in the target learning group based on the first image sequence recognition by the first sub-model of the learning status detection network model, and obtaining learning behavior recognition results of each learner in the target learning group based on the second image sequence recognition by the second sub-model of the learning status detection network model.

[0056] S104, determining a learning state detection result of each learner in the target learning group at a corresponding moment based on the facial expression recognition result and the learning behavior recognition result.

[0057] In some embodiments, in S101, after obtaining the first video information of the target learning group captured by the RGB camera and the second video information of the target learning group captured by the dynamic vision camera, the method further includes:

[0058] Decomposing the first video information into a first sub-video set consisting of a plurality of sub-video segments in time sequence, and decomposing the second video information into a second sub-video set consisting of a plurality of sub-video segments in time sequence; wherein in the same sub-video set, there is an overlapping segment of a preset time length between every two sub-videos that are adjacent to each other in time sequence;

[0059] Extracting a preset number of video frames from the first sub-video set based on a preset time interval to obtain a first video frame set, and extracting a preset number of video frames from the second sub-video set based on a preset time interval to obtain a second video frame set;

[0060] The step of obtaining a first image sequence based on the first video information processing and obtaining a second image sequence based on the second video information processing includes:

[0061] The first image sequence is obtained based on the first video frame set, and the second image sequence is obtained based on the second video frame set.

[0062] Specifically, the first video information of the learner's facial expression collected by the RGB camera and the DVS dynamic vision camera device, namely the RGB video data V RGB and the second video information of the learning behavior, namely the DVS video data V DVS Decomposed into a set of multiple overlapping sub-video segments in chronological order, which can be represented as a video segment set Set corresponding to the first video information RGB = {V RGB1 ,V RGB2 ,…,V RGB17} and the video segment set Set corresponding to the second video information DVS = {V DVS1 ,V DVS2 ,…,V DVS17}, where each sub-video is 20 seconds long, and the overlap time between the current sub-video and the previous sub-video can be set to 10 seconds.

[0063] Furthermore, the RGB sub-video set Set RGB All sub-videos in the DVS can be processed into frames to obtain a first video frame set. DVS All sub-videos in the video are processed by frame division to obtain a second video frame set; wherein, one frame is extracted from each sub-video at an interval of 800 milliseconds, and 16 frames are extracted in total.

[0064] Optionally, the frame set from the same RGB sub-video can be stored in a folder file named "time-learner-expression type" RGB , the frame sets from the same DVS sub-video are stored in a folder named "time-learner-behavior type" DVS ,The learners can be identified based on face recognition technology, or can be set according to the ,collection target preset by the camera, that is, one camera is aimed at one learner.

[0065] In some embodiments, in S102, face detection may be performed on each video frame in the first video information, and the video frames in which faces are recognized may be cropped, and facial image regions of all the video frames in which faces are recognized may be retained;

[0066] Performing denoising processing on each video frame in the second video information, and outputting the denoised video frame;

[0067] The facial image area and the denoised video frame are respectively cropped to a preset pixel size.

[0068] Specifically, for file RGB The RGB images in the folder are detected and cropped using the Dlib toolbox, retaining only the facial image area, and scaled to 400×400 pixels. DVS The DVS images in the folder are denoised using the MaskedDenoising network to ensure that the DVS images can remove irrelevant background noise and retain useful dynamic visual information.

[0069] Furthermore, you can DVS DVS images and files in folders RGB The RGB images in the folder are enhanced by the following methods: first, the images are rotated 45 degrees counterclockwise with a probability of 50% and strongly exposed, and then the images are cropped to 224 × 224 pixels by center cropping.

[0070] In some embodiments, in S103, Figure 2 As shown, the facial expression recognition result of each learner in the target learning group is obtained based on the first image sequence recognition through the first sub-model of the learning state detection network model, the first sub-model is the fine expression recognition model FVATrans, and the fine expression recognition model FVATrans includes a temporal modeling network, a facial change feature fusion network and a fully connected layer. The specific application process includes:

[0071] The first image sequence composed of the input RGB images, that is, the facial expression video sequence X, is sent to the backbone network for feature extraction to obtain the facial spatial features corresponding to each frame of the image, as shown below:

[0072] F i =SwinT(X i ),i∈{1,2,…,16};

[0073] Among them, SwinT(·) means that SwinTransformer is used as the backbone network for feature extraction, F i ∈R C ×H×W represents the facial spatial features corresponding to the i-th RGB image in the video sequence, where C represents the number of channels, specifically 512, and H and W represent the height and width of the feature map, respectively, both of which are 7.

[0074] Then, the learner's facial spatial feature sequence F is further convolved to obtain a static facial spatial feature sequence F S , subtract the facial spatial features corresponding to adjacent frames and perform convolution processing to obtain the dynamic facial spatial feature sequence F D .

[0075]

[0076] Among them, Conv-D(·) represents the convolution operation with stride 2, and F i S represents the static spatial features corresponding to the i-th image frame, It represents the dynamic spatial features corresponding to the i-th image frame. The dimensions of both spatial features are 256×4×4.

[0077] Further, the obtained static facial spatial feature sequence F S and dynamic facial spatial feature sequence F D Flatten into a one-dimensional tensor to obtain preliminary static facial spatial features I s and dynamic facial spatial features I D , the element dimensions in both sequences are C, which can be expressed as:

[0078]

[0079] in It is a squeezing operation that involves point-wise convolution and flattening.

[0080] Furthermore, the obtained static facial spatial feature I s and Dynamic Facial Space I D Add the time sequence position code respectively and apply the formula:

[0081]

[0082] in, represents the temporal embedding of the i-th static space Token, Represents the temporal embedding of the i-th dynamic space Token.

[0083] Furthermore, the static facial spatial feature sequence [static] and the dynamic facial spatial feature sequence [dynamic] are concatenated with the facial variation spatial feature sequence [variation] to form X 0 As input:

[0084] X 0 ={[static],[dynamic],[variation]};

[0085] Among them, all elements in [variation] are learnable tensors, and there are 5 facial variation Token tensors in total.

[0086] Then X 0 After the Transformer processing, the final facial variation spatial feature sequence [variation] is obtained, while the static facial spatial feature sequence [static] and the dynamic facial spatial feature sequence [dynamic] are ignored in the subsequent steps. The Transformer consists of multiple Transformer encoding blocks, and the process of each Transformer block can be expressed as:

[0087]

[0088] Among them, MSA(·) stands for multi-head self-attention, MLP(·) stands for multi-layer perceptron, and LN(·) stands for layer normalization. After the last layer of Transformer, the facial variation token sequence [variation] is selected as the output of Transformer.

[0089] Furthermore, the facial variation spatial feature sequence [variation] output by the temporal modeling network is input into the feature encoding network to enhance its nonlinear representation. The feature encoding network output result is used as the preliminary facial variation spatial feature sequence:

[0090] [variation feature]={M1,M2,…,M N}

[0091] It can be expressed as:

[0092] M i =σ(FC(Mi )),i∈{1,2,…,5};

[0093] Among them, FC(·) represents a fully connected layer operation, σ(·) represents the ReLU activation function, and M i represents the i-th facial variation feature.

[0094] Through a relationship matrix G = [w1, w2, ... w5] ∈ R 5×5 To represent the relationship between facial variation features in a facial expression sequence. The relationship ω(j,m) between the jth facial variation feature and the mth facial variation feature is obtained by calculating the Euclidean distance between the two and using the Tanh activation function to map the distance value to the range of 0 to 1. The formula is:

[0095]

[0096] in, Represents the Tanh activation function.

[0097] Furthermore, the relationship vector corresponding to the j-th facial change feature is multiplied by other facial change features to obtain the j-th relationship perception feature, which can be expressed as:

[0098]

[0099] in, represents the jth relation-aware feature.

[0100] Combining the jth facial feature with the jth relationship-aware feature, we obtain the final jth facial change feature, which can be expressed as:

[0101]

[0102] Among them, δ represents a hyperparameter used to balance facial features and relationship-aware features, and its value here is 0.5. After the above operation, a set of final facial change feature sequences can be obtained, which contains a total of 5 facial change features.

[0103] Furthermore, the facial change features in the facial change feature sequence y output by the facial change feature fusion network are averaged to obtain the discriminative feature representation, and the formula is applied:

[0104]

[0105] Among them, F Expression It is a discriminative feature representation.

[0106] Finally, the discriminative features obtained are represented as F Expression Send it to the fully connected layer to obtain the final recognition result, and apply the formula:

[0107] Exp prediction =FC(expression_feature);

[0108] Among them, FC(·) represents the fully connected layer network, and the expression prediction result tensor Exp prediction ∈R J , J represents the category of expression recognition.

[0109] Exemplarily, the recognized expression categories may include four categories: strong happiness, happiness, neutrality, and frustration, which is not limited in the present application.

[0110] It should be noted that, during the training process of the first sub-model of the learning state detection network model, in order to minimize the error between the predicted expression category and the real expression category, the cross entropy loss function and the Adam optimizer can be used to optimize the fine facial expression recognition network, and the initial learning rate is set to 0.01. After 15 epoch training rounds, the learning rate is reduced by the cosine annealing algorithm to make the network learn more stably until the loss value no longer decreases.

[0111] In some embodiments, in S103, Figure 3 As shown, obtaining the learning behavior recognition result of each learner in the target learning group based on the second image sequence recognition through the second sub-model of the learning state detection network model includes the steps of:

[0112] In some embodiments, Figure 3 As shown in the figure, the learning behavior understanding model FIMFNet includes a text encoding network and a visual encoding network; wherein the text encoding network is composed of 12 Transformer encoders, each of which constitutes a Transformer layer; the visual encoder network is composed of a frame interaction Transformer network and a memory fusion Transformer network; the text encoding network is used to convert the input learning behavior text into a fixed-length feature vector representation. The visual encoding network will process and encode the DVS image data during the learning process; the frame interaction Transformer network can capture the temporal dependency between image frames, while the memory fusion Transformer network can fuse and integrate the information in multiple image frames.

[0113] In some embodiments, in the second sub-model of the learning state detection network model, by setting relevant texts of preset types of concentrated learning behaviors, such as problem-solving behavior, reading behavior, active discussion behavior, inattention behaviors such as playing with mobile phones, and quarreling behavior, etc., various descriptions of relevant texts of learning behaviors can be processed into a sequence of relevant text descriptions.

[0114] Taking five learning behaviors as an example, the relevant text description sequence X={x1,x2,…,x5} corresponding to the input five learning behaviors is then converted into the corresponding vector representation [word]={w1,w2,…,w5} through the embedding layer, which can be expressed as:

[0115] w i =Embedding(x i ),i∈{1,2,…,5};

[0116] Furthermore, the text description vector sequence [word] is used as input and processed through multiple Transformer layers to obtain the final learning behavior features, including:

[0117] When the input text description vector sequence enters a Transformer layer, it is first normalized. The specific application formula is:

[0118]

[0119] Among them, a l is the original output of layer l, is the result after layer normalization.

[0120] Then, the calculation is performed based on the single-head self-attention mechanism to obtain the single-head self-attention result. The specific application formula is:

[0121]

[0122] Among them, W Q ,W K ,W V is a parameter, h is the number of self-attention heads, SA(T l ) is the result of single-head self-attention in layer l.

[0123] Furthermore, after obtaining the single-head self-attention result, calculate the multi-head self-attention result and apply the formula:

[0124] MSA(T)=[SA1(T);SA2(T);SA3(T);…,SA h (T)]W P ;

[0125] There is a residual connection between the result of multi-head self-attention and the original input, and the formula is as follows:

[0126] z l ′ =MSA(LN(z l-1 ))+z l-1 ;

[0127] Finally, a layer normalization operation is performed and a fully connected neural network is passed through. There is also a residual connection between the fully connected neural network result and the original input. The process is described as follows:

[0128] z l =MLP(LN(z l ′ ))+z l ′ ;

[0129] After 12 Transformer layers, the final learning behavior text description feature sequence [action] = {act1, act2, ..., act4} is obtained, where act i Represents the text description feature corresponding to the i-th learning behavior, thereby processing the learning behavior description text into a learning behavior text description feature sequence [action].

[0130] In some embodiments, during the application of the second sub-model of the learning state detection network model, the learning behavior video discriminative visual features reflecting the learning behavior category can be obtained based on the second image sequence, that is, the DVS learning behavior video frame sequence, and the learning behavior video discriminative visual features are further compared with the preset learning behavior text description feature sequence [action], so as to determine the learning behavior recognition result of the corresponding learner, specifically including:

[0131] The input DVS image sequence V can be processed in blocks, and each image can be divided into 196 non-overlapping image blocks.

[0132] It can be understood that the tth frame is divided into 196 non-overlapping blocks Each block has a size of 16×16 pixels, where t∈{1,2,…,16}. Image Block

[0133] Then, through linear projection And add position encoding to transform it into a visual feature sequence.

[0134] Furthermore, a learnable [class]Token tensor x is added before the visual feature sequence class , the state of this Token at the encoder output is used as the representation of the frame.

[0135] Specifically, at frame t, the input of the frame interaction Transformer It is expressed as:

[0136]

[0137] Among them, e spa Represents spatial position encoding, Ex t,1 Represents the image block corresponding to frame t.

[0138] Further, the Token sequence Input into a 10-layer frame interaction Transformer to obtain the frame-level representation h t , the processing of each layer remains consistent. Specifically, an interaction token is first introduced for each frame to extract, send and receive information, so that visual information can be exchanged between frames.

[0139] Specifically, the interaction token of the tth frame of the lth layer—— It is through [class]Token—— It is obtained by linear transformation, so that the interaction Token can extract the visual information of the current frame.

[0140] Furthermore, a multi-head self-attention mechanism is used to learn the global spatiotemporal dependencies of the input video for all interaction tokens. This process in the lth layer can be expressed as:

[0141]

[0142] in, And LN(·) represents layer normalization, and MSA(·) represents the multi-head self-attention mechanism.

[0143] In some embodiments, visual tokens, [class] tokens and their associated interaction tokens can be used together to learn visual representations, while the interaction tokens involved are used to diffuse global spatiotemporal dependencies for learning, applying the formula:

[0144]

[0145] Among them, ([·,·]) represents the feature concatenation of frame tokens and interaction tokens, LN(·) represents layer normalization, and MSA(·) represents multi-head self-attention operation.

[0146] The interaction token is then discarded to ensure that the interaction token is only used for frame-to-frame interaction within each layer and is not passed to the next layer. The feedforward network then operates on the frame token to obtain:

[0147]

[0148] Where LN(·) represents layer normalization and FFN(·) represents feed-forward neural network.

[0149] Furthermore, the [class]Token corresponding to each frame in the final output of the frame interaction transformer is taken out, and the [class]Token corresponding to the frames of the same video clip are sequenced, and then the sequence is temporally embedded to obtain the input of the memory fusion transformer, which can be expressed as:

[0150]

[0151] in, Indicates that the tth frame passes through L c The [class]Token corresponding to the frame interaction Transformer processing of the layer, and H is the input sequence of the memory fusion Transformer.

[0152] Then, the input H is processed through four Transformer encoding blocks to obtain the final result. The processing process of each Transformer encoding block can be expressed as:

[0153]

[0154] Among them, MSA(·) represents the multi-head self-attention mechanism, MLP(·) represents the multi-layer perceptron operation, and LN(·) represents layer normalization.

[0155] Perform an average pooling operation on the output of the last Transformer encoding block to obtain the discriminative visual feature representation v of the learned behavior video:

[0156]

[0157] Among them, AvgPool(·) represents the average pooling operation, Represents the output result of the frame interaction Transformer.

[0158] The cosine similarity between the discriminative visual features of the learning behavior video and each text description feature in a preset learning behavior text description feature sequence is calculated, and the learning behavior category corresponding to the text description feature with the highest cosine similarity is output as the learning behavior recognition result of the corresponding learner.

[0159] In some embodiments, in S104, determining the learning state detection result of each learner in the target learning group at a corresponding moment based on the facial expression recognition result and the learning behavior recognition result includes:

[0160] Determine the distribution of facial expression recognition results for each learner, and determine the facial expression parameters of the target learning group based on the facial expression category with the largest proportion in the distribution of facial expression recognition results;

[0161] Determine the distribution of learning behavior identification results for each learner, and determine the learning behavior parameters of the target learning group based on the learning behavior category with the largest proportion in the distribution of learning behavior identification results;

[0162] The learning state detection result of each learner in the target learning group at a corresponding moment is obtained by weighted calculation based on the facial expression parameters and the learning behavior parameters.

[0163] In some embodiments, as shown in Table 1 below, the learning status detection result of the target learning group at the corresponding time can be determined based on the content of the following table to determine the mapping relationship between the facial expression parameters, the learning behavior parameters and the learning status:

[0164] Table 1

[0165]

[0166]

[0167] In some embodiments, the teacher end or the learning management end can monitor the learning status of the learning group in real time, and remind and adjust the learning group with poor learning status. This embodiment of the present application is not limited to this.

[0168] The following are device embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0169] See next Figure 4 , which is a structural diagram of a learning state detection device based on cross-modal representation learning provided by an exemplary embodiment of the present application. The device can be implemented as all or part of a terminal through software, hardware, or a combination of both, and can also be integrated on a server as an independent module. The learning state detection device based on cross-modal representation learning in the embodiment of the present application can be applied to a terminal or a cloud. The device 40 includes a data acquisition module 401, an image processing module 402, and a detection module 403, wherein:

[0170] The data acquisition module 401 is used to obtain the first video information of the target learning group taken by the RGB camera and the second video information of the target learning group taken by the dynamic vision camera;

[0171] The image processing module 402 is used for processing the first video information to obtain a first image sequence, and processing the second video information to obtain a second image sequence;

[0172] The detection module 403 is used to input the first image sequence and the second image sequence into a trained learning state detection network model, obtain the facial expression recognition result of each learner in the target learning group based on the first image sequence recognition through the first sub-model of the learning state detection network model, and obtain the learning behavior recognition result of each learner in the target learning group based on the second image sequence recognition through the second sub-model of the learning state detection network model;

[0173] The detection module 403 is further used to determine the learning state detection result of each learner in the target learning group at a corresponding moment according to the facial expression recognition result and the learning behavior recognition result.

[0174] It should be noted that when the device 40 provided in the above embodiment executes the learning state detection method based on cross-modal representation learning, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device provided in the above embodiment and the learning state detection method embodiment based on cross-modal representation learning belong to the same concept. The implementation process is detailed in the method embodiment, which will not be repeated here.

[0175] An embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the program.

[0176] See also Figure 5 , which is a structural block diagram of an electronic device provided in an embodiment of the present application.

[0177] like Figure 5 As shown, the electronic device 500 includes: a processor 501 and a memory 502 .

[0178] In the embodiment of the present application, the processor 501 is the control center of the computer system, which can be a processor of a physical machine or a processor of a virtual machine. The processor 501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 501 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array).

[0179] The processor 501 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also called a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state.

[0180] The memory 502 may include one or more computer-readable storage media, which may be non-transitory. The memory 502 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments of the present application, the non-transitory computer-readable storage medium in the memory 502 is used to store at least one instruction, which is used to be executed by the processor 501 to implement the method in the embodiment of the present application.

[0181] In some embodiments, the electronic device 500 further includes: a peripheral device interface 503 and at least one peripheral device 504. The processor 501, the memory 502 and the peripheral device interface 503 can be connected via a bus or a signal line. Each peripheral device 504 can be connected to the peripheral device interface 503 via a bus, a signal line or a circuit board. Specifically, the peripheral device 504 includes: a display screen, a camera and an audio circuit. The peripheral device interface 503 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 501 and the memory 502.

[0182] In some embodiments of the present application, the processor 501, the memory 502, and the peripheral device interface 503 are integrated on the same chip or circuit board; in some other embodiments of the present application, any one or two of the processor 501, the memory 502, and the peripheral device interface 503 can be implemented on a separate chip or circuit board. This embodiment of the present application does not specifically limit this.

[0183] The electronic device structure block diagram shown in the embodiment of the present application does not constitute a limitation on the electronic device 500. The electronic device 500 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0184] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the method of any of the above embodiments are implemented. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0185] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution can be essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A learning state detection method based on cross-modal representation learning, characterized in that: include: Acquire first video information of a target learning group captured by an RGB camera and second video information of the target learning group captured by a dynamic vision camera; Processing the first video information to obtain a first image sequence, and processing the second video information to obtain a second image sequence; Inputting the first image sequence and the second image sequence into a trained learning state detection network model, obtaining a facial expression recognition result of each learner in the target learning group through the first sub-model of the learning state detection network model based on the first image sequence recognition, and obtaining a learning behavior recognition result of each learner in the target learning group through the second sub-model of the learning state detection network model based on the second image sequence recognition; The learning state detection result of each learner in the target learning group at a corresponding moment is determined based on the facial expression recognition result and the learning behavior recognition result.

2. The method according to claim 1, characterized in that After obtaining the first video information of the target learning group photographed by the RGB camera and the second video information of the target learning group photographed by the dynamic vision camera, the method further includes: Decomposing the first video information into a first sub-video set consisting of a plurality of sub-video segments in time sequence, and decomposing the second video information into a second sub-video set consisting of a plurality of sub-video segments in time sequence; wherein in the same sub-video set, there is an overlapping segment of a preset time length between every two sub-videos that are adjacent to each other in time sequence; Extracting a preset number of video frames from the first sub-video set based on a preset time interval to obtain a first video frame set, and extracting a preset number of video frames from the second sub-video set based on a preset time interval to obtain a second video frame set; The step of obtaining a first image sequence based on the first video information processing and obtaining a second image sequence based on the second video information processing includes: The first image sequence is obtained based on the first video frame set, and the second image sequence is obtained based on the second video frame set.

3. The method according to claim 1, characterized in that Before obtaining a first image sequence based on the first video information and obtaining a second image sequence based on the second video information, the method further includes: Performing face detection on each video frame in the first video information, cropping the video frames in which faces are recognized, and retaining the facial image areas of all the video frames in which faces are recognized; Performing denoising processing on each video frame in the second video information, and outputting the denoised video frame; The facial image area and the denoised video frame are respectively cropped to a preset pixel size.

4. The method according to claim 1, characterized in that: The method of obtaining the facial expression recognition result of each learner in the target learning group based on the first image sequence recognition by the first sub-model of the learning state detection network model includes: Performing feature extraction on each image frame in the first image sequence to obtain facial spatial features corresponding to each image frame; The facial spatial features are processed to obtain a static facial spatial feature sequence, and facial spatial features corresponding to adjacent image frames are subtracted and processed to obtain a dynamic facial spatial feature sequence; Obtaining a facial change spatial feature sequence based on the static facial spatial feature sequence and the dynamic facial spatial feature sequence; Each facial change feature is extracted based on the facial change spatial feature sequence, an expression prediction result tensor is obtained based on the average value of each facial change feature, and an expression category label of the corresponding learner is determined based on the expression prediction result tensor.

5. The method according to claim 1, characterized in that The obtaining of the learning behavior recognition result of each learner in the target learning group based on the second image sequence recognition by the second sub-model of the learning state detection network model includes: Dividing each image frame in the second image sequence into a plurality of image blocks, and adding a position code to each of the image blocks to obtain a visual feature sequence; Processing the visual feature sequence sequentially through the frame interaction transformer module and the memory fusion transformer module of the second sub-model to obtain discriminative visual features of the corresponding learning behavior video; The cosine similarity between the discriminative visual features of the learning behavior video and each text description feature in a preset learning behavior text description feature sequence is calculated, and the learning behavior category corresponding to the text description feature with the highest cosine similarity is output as the learning behavior recognition result of the corresponding learner.

6. The method according to any one of claims 1 to 5, characterized in that: The step of determining the learning state detection result of each learner in the target learning group at a corresponding moment based on the facial expression recognition result and the learning behavior recognition result comprises: Determine the distribution of facial expression recognition results for each learner, and determine the facial expression parameters of the target learning group based on the facial expression category with the largest proportion in the distribution of facial expression recognition results; Determine the distribution of learning behavior identification results for each learner, and determine the learning behavior parameters of the target learning group based on the learning behavior category with the largest proportion in the distribution of learning behavior identification results; The learning status detection result of the target learning group at the corresponding moment is obtained by weighted calculation based on the facial expression parameters and the learning behavior parameters.

7. A learning state detection device based on cross-modal representation learning, characterized in that: include: A data acquisition module, used for acquiring first video information of a target learning group photographed by an RGB camera and second video information of the target learning group photographed by a dynamic vision camera; An image processing module, configured to obtain a first image sequence based on the first video information and obtain a second image sequence based on the second video information; A detection module, used for inputting the first image sequence and the second image sequence into a trained learning state detection network model, obtaining a facial expression recognition result of each learner in the target learning group through a first sub-model of the learning state detection network model based on the first image sequence recognition, and obtaining a learning behavior recognition result of each learner in the target learning group through a second sub-model of the learning state detection network model based on the second image sequence recognition; The detection module is also used to determine the learning state detection result of each learner in the target learning group at a corresponding moment according to the facial expression recognition result and the learning behavior recognition result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images