A learning state detection method based on cross-modal representation learning

Abstract

The embodiment of the application discloses a learning state detection method based on cross-modal representation learning, relates to the technical field of teaching informatization, and comprises the following steps: acquiring first video information collected by an RGB camera and second video information collected by a dynamic visual camera, respectively processing the first video information and the second video information to obtain a first image sequence and a second image sequence, inputting the first image sequence and the second image sequence into a trained learning state detection network model, respectively identifying facial expression recognition results and learning behavior recognition results of each learner in a target learning group, and determining learning state detection results of each learner in the target learning group at corresponding moments. According to the method, the overall learning of the learning group is estimated based on the facial expression recognition results and the learning behavior recognition results of each learner, the collective learning effect of the learners can be accurately evaluated, and teachers can know the real-time learning state of each learning group in real time.

Description

Technical Field

[0001] This application relates to the field of educational information technology, and in particular to a learning state detection method based on cross-modal representation learning. Background Technology

[0002] Students can improve their learning efficiency by forming study groups or teams to learn collaboratively, sharing goals, motivating each other, and jointly adjusting learning strategies. With the development of educational technology, traditional classroom teaching models are gradually shifting towards intelligent and personalized learning environments. In this context, assessing the learning status of individual students within study groups and teams becomes more difficult, making it hard to determine whether each student is engaged in learning, presenting multiple challenges.

[0003] Traditional methods of assessing learning status rely on the subjective judgment of teachers and learners, which can lead to problems such as excessive subjectivity and untimely assessment. Furthermore, unlike traditional learning environments, in learning groups and teams, the learning unit is the corresponding learning group rather than an individual student. Therefore, this change in the assessment focus presents a greater challenge to the perception of learning status in team learning.

[0004] With the deepening integration of information technology and education, smart classrooms, supported by technology, play an important role in optimizing learning outcomes. Compared to traditional classrooms, smart classrooms offer convenience for collecting learners' learning behaviors; however, in the process of group learning, problems such as inaccurate and untimely perception of the learning status of learning groups still exist.

[0005] Therefore, there is a need for an auxiliary means to improve the quality of learning in the team learning process and enhance students' team learning collaboration ability. Summary of the Invention

[0006] This application provides a learning state detection method based on cross-modal representation learning to address the shortcomings of the aforementioned related technologies. The technical solution is as follows:

[0007] In a first aspect, embodiments of this application provide a learning state detection method based on cross-modal representation learning, including:

[0008] Acquire the first video information of the target learning group captured by the RGB camera and the second video information of the target learning group captured by the dynamic vision camera;

[0009] A first image sequence is obtained by processing the first video information, and a second image sequence is obtained by processing the second video information.

[0010] The first image sequence and the second image sequence are input into the trained learning state detection network model. The first sub-model of the learning state detection network model identifies the facial expression recognition result of each learner in the target learning group based on the first image sequence. The second sub-model of the learning state detection network model identifies the learning behavior recognition result of each learner in the target learning group based on the second image sequence.

[0011] Based on the facial expression recognition results and the learning behavior recognition results, the learning state detection results of each learner in the target learning group at the corresponding time are determined.

[0012] In one alternative embodiment of the first aspect, after acquiring the first video information of the target learning group captured by the RGB camera and the second video information of the target learning group captured by the dynamic vision camera, the method further includes:

[0013] The first video information is decomposed into a first sub-video set consisting of multiple sub-video segments in chronological order, and the second video information is decomposed into a second sub-video set consisting of multiple sub-video segments in chronological order; wherein, in the same sub-video set, every two sub-videos that are adjacent in chronological order have an overlapping segment of a preset duration.

[0014] A first video frame set is obtained by extracting a preset number of video frames from the first sub-video set based on a preset time interval, and a second video frame set is obtained by extracting a preset number of video frames from the second sub-video set based on a preset time interval.

[0015] The process of obtaining a first image sequence based on the first video information and obtaining a second image sequence based on the second video information includes:

[0016] The first image sequence is obtained by processing the first set of video frames, and the second image sequence is obtained by processing the second set of video frames.

[0017] In one alternative embodiment of the first aspect, before processing the first image sequence based on the first video information and processing the second image sequence based on the second video information, the method further includes:

[0018] For each video frame in the first video information, perform face detection, crop the video frames in which faces are identified, and retain the facial image regions of all video frames in which faces are identified.

[0019] Denoise the video frame in the second video information and output the denoised video frame.

[0020] The facial image region and the denoised video frame are respectively cropped to a preset pixel size.

[0021] In one alternative embodiment of the first aspect, the step of obtaining the facial expression recognition result of each learner in the target learning group based on the first image sequence recognition by the first sub-model of the learning state detection network model includes:

[0022] Feature extraction is performed on each image frame in the first image sequence to obtain the facial spatial features corresponding to each image frame;

[0023] The facial spatial features are processed to obtain a static facial spatial feature sequence, and the facial spatial features corresponding to adjacent image frames are subtracted and processed to obtain a dynamic facial spatial feature sequence.

[0024] A facial change spatial feature sequence is obtained by processing the static facial spatial feature sequence and the dynamic facial spatial feature sequence.

[0025] Each facial change feature is extracted based on the spatial feature sequence of facial changes. An expression prediction result tensor is obtained based on the mean value of each facial change feature. The expression category label of the corresponding learner is determined based on the expression prediction result tensor.

[0026] In one alternative embodiment of the first aspect, the step of obtaining the learning behavior recognition result of each learner in the target learning group based on the second image sequence recognition by the second sub-model of the learning state detection network model includes:

[0027] Each image frame in the second image sequence is divided into multiple image blocks, and a positional encoding is added to each image block to obtain a visual feature sequence;

[0028] The visual feature sequence is processed sequentially through the frame interaction transformer module and the memory fusion transformer module of the second sub-model to obtain the discriminative visual features of the corresponding learning behavior video.

[0029] Calculate the cosine similarity between the discriminative visual features of the learning behavior video and each text description feature in the preset learning behavior text description feature sequence, and output the learning behavior category corresponding to the text description feature with the highest cosine similarity as the learning behavior recognition result of the corresponding learner.

[0030] In one alternative embodiment of the first aspect, the step of determining the learning state detection result of each learner in the target learning group at a corresponding time based on the facial expression recognition result and the learning behavior recognition result includes:

[0031] Determine the distribution of facial expression recognition results for each learner, and determine the facial expression parameters of the target learning group based on the facial expression category that accounts for the largest proportion in the distribution of facial expression recognition results;

[0032] Determine the distribution of learning behavior recognition results for each learner, and determine the learning behavior parameters of the target learning group based on the learning behavior category that accounts for the largest proportion in the distribution of learning behavior recognition results;

[0033] The learning status detection result of the target learning group at the corresponding time is obtained by weighted calculation based on the facial expression parameters and the learning behavior parameters.

[0034] Secondly, embodiments of this application also provide a learning state detection device based on cross-modal representation learning, comprising:

[0035] The data acquisition module is used to acquire the first video information of the target learning group captured by the RGB camera and the second video information of the target learning group captured by the dynamic vision camera;

[0036] The image processing module is used to process the first video information to obtain a first image sequence and to process the second video information to obtain a second image sequence.

[0037] The detection module is used to input the first image sequence and the second image sequence into a trained learning state detection network model, and to obtain the facial expression recognition result of each learner in the target learning group based on the first image sequence through the first sub-model of the learning state detection network model, and to obtain the learning behavior recognition result of each learner in the target learning group based on the second image sequence through the second sub-model of the learning state detection network model.

[0038] The detection module is further configured to determine the learning status detection result of each learner in the target learning group at a corresponding time based on the facial expression recognition result and the learning behavior recognition result. Thirdly, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the method provided by the first aspect or any implementation thereof of the embodiments of this application.

[0039] Fourthly, this application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method provided by the first aspect of the embodiments of this application or any implementation thereof.

[0040] The beneficial effects of the technical solutions provided in some embodiments of this application include at least the following:

[0041] This application provides a learning state detection method based on cross-modal representation learning. It estimates the overall learning of the learning group based on the facial emotion recognition results and learning behavior recognition results of each learner. This method can accurately evaluate the collective learning effect of learners and helps teachers understand the real-time learning status of each learning group. Based on the real-time learning status, teachers can adjust the learning methods of each learning group or learning team and improve the learning efficiency of the team. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating a learning state detection method based on cross-modal representation learning according to an embodiment of this application.

[0044] Figure 2 This is a flowchart illustrating a learning state detection method based on cross-modal representation learning according to an embodiment of this application.

[0045] Figure 3 This is a flowchart illustrating a learning state detection method based on cross-modal representation learning according to an embodiment of this application.

[0046] Figure 4 This is a schematic diagram of the structure of a learning state detection device based on cross-modal representation learning provided in an embodiment of this application;

[0047] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or modules is not limited to the steps or modules listed, but may optionally include steps or modules not listed, or may optionally include other steps or modules inherent to such process, method, product, or apparatus.

[0050] It should be noted that the terms "first" and "second" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permitted. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in an order other than those described or illustrated herein.

[0051] The present application will now be described in detail with reference to specific embodiments.

[0052] Next, combine Figure 1 This paper introduces a learning state detection method based on cross-modal representation learning provided in the embodiments of this application. For details, please refer to... Figure 1 , Figure 1 The illustration shows a flowchart of a learning state detection method based on cross-modal representation learning according to an embodiment of this application. The method includes the following steps:

[0053] S101, acquire the first video information of the target learning group captured by the RGB camera and the second video information of the target learning group captured by the dynamic vision camera.

[0054] S102, a first image sequence is obtained by processing the first video information, and a second image sequence is obtained by processing the second video information.

[0055] S103, the first image sequence and the second image sequence are input into the trained learning state detection network model. The first sub-model of the learning state detection network model identifies the facial expression recognition result of each learner in the target learning group based on the first image sequence. The second sub-model of the learning state detection network model identifies the learning behavior recognition result of each learner in the target learning group based on the second image sequence.

[0056] S104, Based on the facial expression recognition result and the learning behavior recognition result, determine the learning state detection result of each learner in the target learning group at the corresponding time.

[0057] In some embodiments, after obtaining the first video information of the target learning group captured by the RGB camera and the second video information of the target learning group captured by the dynamic vision camera in S101, the method further includes:

[0058] The first video information is decomposed into a first sub-video set consisting of multiple sub-video segments in chronological order, and the second video information is decomposed into a second sub-video set consisting of multiple sub-video segments in chronological order; wherein, in the same sub-video set, every two sub-videos that are adjacent in chronological order have an overlapping segment of a preset duration.

[0059] A first video frame set is obtained by extracting a preset number of video frames from the first sub-video set based on a preset time interval, and a second video frame set is obtained by extracting a preset number of video frames from the second sub-video set based on a preset time interval.

[0060] The process of obtaining a first image sequence based on the first video information and obtaining a second image sequence based on the second video information includes:

[0061] The first image sequence is obtained by processing the first set of video frames, and the second image sequence is obtained by processing the second set of video frames.

[0062] Specifically, the first video information of the learner's facial expressions, namely RGB video data V, is acquired by the RGB camera and the DVS dynamic vision camera device. RGB The second video information related to learning behavior is the DVS video data V. DVS The set of multiple overlapping sub-video segments, decomposed chronologically, can be represented as a set of video segments corresponding to the first video information. RGB ={V RGB1 V RGB2 ,…,V RGB17} and the set of video segments corresponding to the second video information. DVS ={V DVS1 V DVS2 ,…,V DVS17 Each sub-video has a duration of 20 seconds, and the overlap time between the current sub-video and the previous sub-video can be set to 10 seconds.

[0063] Furthermore, it is possible to set the RGB sub-video collection. RGB The first video frame set is obtained by performing frame segmentation on all sub-videos in the DVS sub-video set. DVS All sub-videos are processed into a second set of video frames; in this set, one frame is extracted from each sub-video at 800 millisecond intervals, for a total of 16 frames.

[0064] Optionally, a set of frames from the same RGB sub-video can be stored in a folder named "file" with the format "time-learner-expression type". RGB Frame sets from the same DVS sub-video are stored in a folder named "file" with the sub-video format "time-learner-behavior type". DVS Learners can be identified using facial recognition technology, or they can be set up according to the preset collection target of the camera, that is, one camera is pointed at one learner.

[0065] In some embodiments, in S102, face detection can be performed on each video frame in the first video information, the video frames in which faces are identified can be cropped, and the facial image regions of all video frames in which faces are identified can be retained.

[0066] Denoise the video frame in the second video information and output the denoised video frame.

[0067] The facial image region and the denoised video frame are respectively cropped to a preset pixel size.

[0068] Specifically, for file RGB The RGB images in the folder underwent face detection and cropping using the Dlib toolbox, retaining only the facial image area, and were then scaled to 400×400 pixels. DVS The DVS images in the folder are denoised using a MaskedDenoising network to ensure that irrelevant background noise is removed while retaining useful dynamic visual information.

[0069] Furthermore, it is possible to modify the file. DVS DVS images and files in the folder RGB The RGB images in the folder undergo relevant data enhancement. First, the images are rotated 45 degrees counterclockwise with a 50% probability and subjected to strong exposure. Then, the images are cropped to 224×224 pixels using a center-cropping method.

[0070] In some embodiments, in S103, such as Figure 2 As shown, the facial expression recognition results of each learner in the target learning group are obtained by the first sub-model of the learning state detection network model based on the first image sequence. The first sub-model is the fine expression recognition model FVATrans, which includes a temporal modeling network, a facial change feature fusion network, and a fully connected layer. The specific application process includes:

[0071] The first image sequence, i.e., the facial expression video sequence X, composed of the input RGB images, is fed into the backbone network for feature extraction to obtain the facial spatial features corresponding to each frame, as shown in the following formula:

[0072] F i =SwinT(X i ), i∈{1,2,…,16};

[0073] Where SwinT(·) indicates that SwinTransformer is used as the backbone network for feature extraction, F i ∈R C ×H×W Let C represent the facial spatial features corresponding to the i-th RGB image in the video sequence, where C represents the number of channels, specifically 512, and H and W represent the height and width of the feature map, respectively, both of which are 7.

[0074] Then, the learner's facial spatial feature sequence F is further processed by convolution to obtain the static facial spatial feature sequence F. S The facial spatial features corresponding to adjacent frames are subtracted and then convolutional to obtain the dynamic facial spatial feature sequence F. D .

[0075]

[0076] Where Conv-D(·) represents a lower convolution operation with a stride of 2, F i S This represents the static spatial features corresponding to the i-th image frame. This represents the dynamic spatial feature corresponding to the i-th image frame, and both spatial features have a dimension of 256×4×4.

[0077] Furthermore, the obtained static facial spatial feature sequence F S and dynamic facial spatial feature sequence F D Flattened into a one-dimensional tensor, preliminary static facial spatial features (I) are obtained. s and dynamic facial spatial features I D The elements in both sequences have a dimension of C, which can be represented as:

[0078]

[0079] in It is a compression operation that involves pointwise convolution and flattening.

[0080] Furthermore, for the obtained static facial spatial features I s and dynamic facial space I D Add timing position codes separately, applying the formula:

[0081]

[0082] in, Represents the temporal embedding of the i-th static space token. Represents the temporal embedding of the i-th dynamic space token.

[0083] Furthermore, the static facial spatial feature sequence and the dynamic facial spatial feature sequence are concatenated with the spatial feature sequence representing facial changes to form X. 0 As input:

[0084] X 0 ={[static],[dynamic],[variation]};

[0085] Among them, all elements in [variation] are learnable tensors, and there are a total of 5 facial variation token tensors.

[0086] Then X 0 After processing by the Transformer, the final facial variation spatial feature sequence is obtained. The static and dynamic facial spatial feature sequences are ignored in subsequent steps. The Transformer consists of multiple Transformer coding blocks, and the process of each Transformer block can be represented as follows:

[0087]

[0088] Here, MSA(·) represents multi-head self-attention, MLP(·) represents multilayer perceptron, and LN(·) represents layer normalization. After the last layer of the Transformer, the facial variation token sequence is selected as the output of the Transformer.

[0089] Furthermore, the facial variation spatial feature sequence output by the temporal modeling network is input into the feature encoding network to enhance its nonlinear representation. The output of the feature encoding network serves as the initial facial variation spatial feature sequence.

[0090] [variation feature]={M1,M2,…,M N}

[0091] It can be represented as:

[0092] M i =σ(FC(M)i ),i∈{1,2,…,5};

[0093] Where FC(·) represents a fully connected layer operation, σ(·) represents the ReLU activation function, and M i This represents the i-th facial change feature.

[0094] Through a relation matrix G = [w1, w2, ..., w5] ∈ R 5×5 This is used to represent the relationship between facial change features in a sequence of facial expressions. The relationship ω(j,m) between the j-th and m-th facial change features is obtained by calculating the Euclidean distance between them and mapping the distance value to the range of 0 to 1 using the Tanh activation function, applying the formula:

[0095]

[0096] in, This represents the Tanh activation function.

[0097] Furthermore, the relationship vector corresponding to the j-th facial change feature is multiplied by other facial change features to obtain the j-th relationship-aware feature, which can be represented as:

[0098]

[0099] in, Let j represent the relation-aware feature.

[0100] Combining the j-th facial feature with the j-th relation-aware feature yields the final j-th facial change feature, which can be represented as:

[0101]

[0102] Here, δ represents the hyperparameter used to balance facial features and relation-aware features, and is set to 0.5. After the above operations, a final set of facial change features can be obtained, which contains a total of 5 facial change features.

[0103] Furthermore, the facial change features in the facial change feature sequence y output by the fusion network are averaged to obtain the discriminative feature representation, and the formula is applied:

[0104]

[0105] Among them, F Expression This represents the discriminative features.

[0106] Finally, the obtained discriminative feature representation F Expression The result is fed into a fully connected layer to obtain the final recognition result, using the formula:

[0107] Exp prediction =FC(expression_feature);

[0108] Where FC(·) represents a fully connected layer network, and the expression prediction result tensor Exp prediction ∈R J J represents the category of facial expression recognition.

[0109] For example, the identified expression categories may include four categories: strong happiness, happiness, neutrality, and depression, which are not limited in this application.

[0110] It should be noted that during the training of the first sub-model of the learning state detection network model, in order to minimize the error between the predicted expression category and the real expression category, the cross-entropy loss function and the Adam optimizer can be used to optimize the fine facial expression recognition network. The initial learning rate is set to 0.01, and after 15 epochs of training, the learning rate is reduced by the cosine annealing algorithm to make the network learn more stably until the loss value no longer decreases.

[0111] In some embodiments, in S103, such as Figure 3 As shown, the learning behavior recognition result of each learner in the target learning group is obtained by the second sub-model of the learning state detection network model based on the second image sequence recognition, including the following steps:

[0112] In some embodiments, such as Figure 3 As shown, the FIMFNet learning behavior understanding model includes a text encoding network and a visual encoding network. The text encoding network consists of 12 Transformer encoders, each forming a Transformer layer. The visual encoder network consists of a frame interaction Transformer network and a memory fusion Transformer network. The text encoding network converts the input learning behavior text into a fixed-length feature vector representation. The visual encoding network processes and encodes the DVS image data during the learning process. The frame interaction Transformer network captures the temporal dependencies between image frames, while the memory fusion Transformer network fuses and integrates information from multiple image frames.

[0113] In some embodiments, in the second sub-model of the learning state detection network model, by setting relevant texts of a preset type of concentrated learning behavior, such as problem-solving behavior, reading behavior, active discussion behavior, inattentive behavior such as playing on mobile phones, and arguing behavior, the various descriptions of relevant texts of learning behavior can be processed into a relevant text description sequence.

[0114] Taking five learning behaviors as an example, the relevant text description sequence X = {x1, x2, ..., x5} corresponding to the five input learning behaviors is then converted into the corresponding vector representation [word] = {w1, w2, ..., w5} through an embedding layer, which can be represented as:

[0115] w i =Embedding(x i ), i∈{1,2,…,5};

[0116] Furthermore, the text description vector sequence [word] is used as input and processed through multiple Transformer layers to obtain the final learned behavioral features, specifically including:

[0117] When the input text description vector sequence enters a Transformer layer, it is first normalized, using the following formula:

[0118]

[0119] Among them, a l This is the original output of the l-th layer. This is the result obtained after layer normalization.

[0120] Then, calculations are performed based on the single-head self-attention mechanism to obtain the single-head self-attention result. The specific application formula is as follows:

[0121]

[0122] Among them, W Q W K W V Here, h is the number of self-attention heads, and SA(T) is the parameter. l ) represents the result of single-head self-attention in the l-th layer.

[0123] Furthermore, after obtaining the single-head self-attention results, the multi-head self-attention results are calculated using the following formula:

[0124] MSA(T)=[SA1(T);SA2(T);SA3(T);…,SA h (T)]W P ;

[0125] The result of multi-head self-attention has a residual connection with the original input, as shown in the following formula:

[0126] z l ′ =MSA(LN(z) l-1 ))+z l-1 ;

[0127] Finally, a layer normalization operation is performed, followed by a fully connected neural network. The result of the fully connected neural network also has residual connections with the original input. This process can be described as follows:

[0128] z l =MLP(LN(z) l ′ ))+z l ′ ;

[0129] The learning process can be passed through 12 Transformer layers to obtain the final sequence of learned behavioral text description features [action] = {act1, act2, ..., act4}, where act1, act2, act3, act4, act4, act4, act5, act6, act7, act8, act9 ... i Let i represent the text description feature corresponding to the i-th learning action. Thus, the text description of the learning action can be processed into a sequence of text description features of the learning action [action].

[0130] In some embodiments, during the application of the second sub-model of the learning state detection network model, discriminative visual features reflecting the learning behavior category can be obtained based on the second image sequence, i.e., the DVS learning behavior video frame sequence. These discriminative visual features are then compared with a preset learning behavior text description feature sequence [action] to determine the learning behavior recognition result of the corresponding learner. Specifically, this includes:

[0131] The input DVS image sequence V can be divided into blocks, with each image consisting of 196 non-overlapping image blocks.

[0132] This can be understood as the t-th frame being divided into 196 non-overlapping blocks. Each block is 16×16 pixels in size, where t∈{1,2,…,16}. Image block

[0133] Then, through linear projection And add positional encoding to convert it into a visual feature sequence.

[0134] Furthermore, a learnable [class]Token tensor x is added before the visual feature sequence. class The state of this token at the encoder output serves as the representation of the frame.

[0135] Specifically, at frame t, the input of the frame interaction Transformer. Represented as:

[0136]

[0137] Among them, e spa Represents spatial location encoding, Ex t,1 This represents the image block corresponding to frame t.

[0138] Furthermore, the token sequence The input is fed into a 10-layer frame-interactive Transformer to obtain a frame-level representation h. t The processing procedure remains consistent across each layer. Specifically, an interaction token is first introduced for each frame to extract, send, and receive information, enabling the exchange of visual information between frames.

[0139] Specifically, the interaction token of the l-th layer, frame t— Through [class]Token—— It is obtained by performing a linear transformation, which enables the interactive token to extract visual information from the current frame.

[0140] Furthermore, a multi-head self-attention mechanism is employed to learn the global spatiotemporal dependencies of the input video across all interaction tokens. This process in layer l can be represented as:

[0141]

[0142] in, Furthermore, LN(·) represents layer normalization, and MSA(·) represents multi-head self-attention mechanism.

[0143] In some embodiments, visual tokens, [class] tokens, and their associated interaction tokens can be used together to learn visual representations, while the interaction tokens involved are used to learn global spatiotemporal dependencies, applying the formula:

[0144]

[0145] Where ([·,·]) represents the feature concatenation of frame token and interaction token, LN(·) represents layer normalization, and MSA(·) represents multi-head self-attention operation.

[0146] The interaction token is then discarded to ensure that it is used only for inter-frame interactions within each layer and is not passed to the next layer. The feedforward network then processes the frame token to obtain:

[0147]

[0148] Where LN(·) represents layer normalization, and FFN(·) represents a feedforward neural network.

[0149] Furthermore, the [class] token corresponding to each frame is extracted from the final output of the frame interaction Transformer, and the [class] tokens corresponding to frames from the same video segment are grouped into a sequence. Then, the sequence is temporally embedded to obtain the input of the memory fusion Transformer, which can be represented as:

[0150]

[0151] in, Indicates that the t-th frame has passed L c The [class]Token corresponding to the frame interaction Transformer of the layer is H, which is the input sequence of the memory fusion Transformer.

[0152] Then, the input H is processed through 4 Transformer encoding blocks to obtain the final result. The processing procedure of each Transformer encoding block can be represented as follows:

[0153]

[0154] In this context, MSA(·) represents multi-head self-attention mechanism, MLP(·) represents multilayer perceptron operation, and LN(·) represents layer normalization.

[0155] Average pooling is performed on the output of the last Transformer encoded block to obtain the discriminative visual feature representation v of the learned behavior video:

[0156]

[0157] Where AvgPool(·) represents the average pooling operation. This indicates the output of the frame interaction Transformer.

[0158] Calculate the cosine similarity between the discriminative visual features of the learning behavior video and each text description feature in the preset learning behavior text description feature sequence, and output the learning behavior category corresponding to the text description feature with the highest cosine similarity as the learning behavior recognition result of the corresponding learner.

[0159] In some embodiments, S104, determining the learning state detection result for each learner in the target learning group at a corresponding time based on the facial expression recognition result and the learning behavior recognition result includes:

[0160] Determine the distribution of facial expression recognition results for each learner, and determine the facial expression parameters of the target learning group based on the facial expression category that accounts for the largest proportion in the distribution of facial expression recognition results;

[0161] Determine the distribution of learning behavior recognition results for each learner, and determine the learning behavior parameters of the target learning group based on the learning behavior category that accounts for the largest proportion in the distribution of learning behavior recognition results;

[0162] The learning status detection result of each learner in the target learning group at the corresponding time is obtained by weighted calculation based on the facial expression parameters and the learning behavior parameters.

[0163] In some embodiments, as shown in Table 1 below, the learning state detection results of the target learning group at the corresponding time can be determined based on the contents of the table, and the mapping relationship between facial expression parameters, learning behavior parameters and learning state can be determined:

[0164] Table 1

[0165]

[0166]

[0167] In some embodiments, the teacher's or learning management terminal can monitor the learning status of the learning group in real time, and remind and adjust the learning group with poor learning status. This application embodiment does not limit this.

[0168] The following are apparatus embodiments of this application, which can be used to execute the method embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of this application.

[0169] Please see below. Figure 4 This is a schematic diagram of a learning state detection device based on cross-modal representation learning, provided as an exemplary embodiment of this application. This device can be implemented as all or part of a terminal through software, hardware, or a combination of both, or it can be integrated as an independent module on a server. The learning state detection device based on cross-modal representation learning in this embodiment can be applied to a terminal or the cloud. The device 40 includes a data acquisition module 401, an image processing module 402, and a detection module 403, wherein:

[0170] The data acquisition module 401 is used to acquire the first video information of the target learning group captured by the RGB camera and the second video information of the target learning group captured by the dynamic vision camera;

[0171] Image processing module 402 is used to process the first video information to obtain a first image sequence and to process the second video information to obtain a second image sequence.

[0172] The detection module 403 is used to input the first image sequence and the second image sequence into the trained learning state detection network model, and to obtain the facial expression recognition result of each learner in the target learning group based on the first image sequence through the first sub-model of the learning state detection network model, and to obtain the learning behavior recognition result of each learner in the target learning group based on the second image sequence through the second sub-model of the learning state detection network model.

[0173] The detection module 403 is also used to determine the learning status detection result of each learner in the target learning group at the corresponding time based on the facial expression recognition result and the learning behavior recognition result.

[0174] It should be noted that the above embodiments of the apparatus 40, when executing the learning state detection method based on cross-modal representation learning, are only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus provided in the above embodiments and the learning state detection method embodiments based on cross-modal representation learning belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0175] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described above.

[0176] Please see Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of this application.

[0177] like Figure 5 As shown, the electronic device 500 includes a processor 501 and a memory 502.

[0178] In this embodiment, the processor 501 is the control center of the computer system, and can be a processor of a physical machine or a processor of a virtual machine. The processor 501 may include one or more processing cores, such as a 4-core processor or an 8-core processor. The processor 501 can be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array).

[0179] Processor 501 may also include a main processor and a coprocessor. The main processor is a processor used to process data in the wake-up state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state.

[0180] Memory 502 may include one or more computer-readable storage media, which may be non-transitory. Memory 502 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments of this application, the non-transitory computer-readable storage media in memory 502 is used to store at least one instruction, which is executed by processor 501 to implement the method in the embodiments of this application.

[0181] In some embodiments, the electronic device 500 further includes a peripheral device interface 503 and at least one peripheral device 504. The processor 501, memory 502, and peripheral device interface 503 can be connected via a bus or signal line. Each peripheral device 504 can be connected to the peripheral device interface 503 via a bus, signal line, or circuit board. Specifically, the peripheral device 504 includes: a display screen, a camera, and audio circuitry. The peripheral device interface 503 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 501 and memory 502.

[0182] In some embodiments of this application, the processor 501, memory 502, and peripheral device interface 503 are integrated on the same chip or circuit board; in other embodiments of this application, any one or two of the processor 501, memory 502, and peripheral device interface 503 can be implemented on separate chips or circuit boards. This application does not specifically limit the implementation in this regard.

[0183] The block diagram of the electronic device shown in the embodiments of this application does not constitute a limitation on the electronic device 500. The electronic device 500 may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0184] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the methods in any of the foregoing embodiments. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0185] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A learning state detection method based on cross-modal representation learning, characterized in that, include: Acquire the first video information of the target learning group captured by the RGB camera and the second video information of the target learning group captured by the dynamic vision camera; A first image sequence is obtained by processing the first video information, and a second image sequence is obtained by processing the second video information. The first image sequence and the second image sequence are input into the trained learning state detection network model. The first sub-model of the learning state detection network model identifies the facial expression recognition result of each learner in the target learning group based on the first image sequence. The second sub-model of the learning state detection network model identifies the learning behavior recognition result of each learner in the target learning group based on the second image sequence. Based on the facial expression recognition results and the learning behavior recognition results, the learning state detection results of each learner in the target learning group at the corresponding time are determined; The process of obtaining facial expression recognition results for each learner in the target learning group based on the first image sequence recognition by the first sub-model of the learning state detection network model includes: Feature extraction is performed on each image frame in the first image sequence to obtain the facial spatial features corresponding to each image frame; The facial spatial features are processed to obtain a static facial spatial feature sequence, and the facial spatial features corresponding to adjacent image frames are subtracted and processed to obtain a dynamic facial spatial feature sequence. A facial change spatial feature sequence is obtained by processing the static facial spatial feature sequence and the dynamic facial spatial feature sequence. Each facial change feature is extracted based on the spatial feature sequence of facial changes. An expression prediction result tensor is obtained based on the mean value of each facial change feature. The expression category label of the corresponding learner is determined based on the expression prediction result tensor. The second sub-model of the learning state detection network model obtains the learning behavior recognition result of each learner in the target learning group based on the second image sequence recognition, including: Each image frame in the second image sequence is divided into multiple image blocks, and a positional encoding is added to each image block to obtain a visual feature sequence; The visual feature sequence is processed sequentially through the frame interaction transformer module and the memory fusion transformer module of the second sub-model to obtain the discriminative visual features of the corresponding learning behavior video. Calculate the cosine similarity between the discriminative visual features of the learning behavior video and each text description feature in the preset learning behavior text description feature sequence, and output the learning behavior category corresponding to the text description feature with the highest cosine similarity as the learning behavior recognition result of the corresponding learner.

2. The method according to claim 1, characterized in that, After acquiring the first video information of the target learning group captured by the RGB camera and the second video information of the target learning group captured by the dynamic vision camera, the method further includes: The first video information is decomposed into a first sub-video set consisting of multiple sub-video segments in chronological order, and the second video information is decomposed into a second sub-video set consisting of multiple sub-video segments in chronological order; wherein, in the same sub-video set, every two sub-videos that are adjacent in chronological order have an overlapping segment of a preset duration. A first video frame set is obtained by extracting a preset number of video frames from the first sub-video set based on a preset time interval, and a second video frame set is obtained by extracting a preset number of video frames from the second sub-video set based on a preset time interval. The process of obtaining a first image sequence based on the first video information and obtaining a second image sequence based on the second video information includes: The first image sequence is obtained by processing the first set of video frames, and the second image sequence is obtained by processing the second set of video frames.

3. The method according to claim 1, characterized in that, Before obtaining the first image sequence based on the first video information and obtaining the second image sequence based on the second video information, the process further includes: For each video frame in the first video information, perform face detection, crop the video frames in which faces are identified, and retain the facial image regions of all video frames in which faces are identified. Denoise the video frame in the second video information and output the denoised video frame. The facial image region and the denoised video frame are respectively cropped to a preset pixel size.

4. The method according to any one of claims 1-3, characterized in that, The step of determining the learning state detection result for each learner in the target learning group at a corresponding time based on the facial expression recognition result and the learning behavior recognition result includes: Determine the distribution of facial expression recognition results for each learner, and determine the facial expression parameters of the target learning group based on the facial expression category that accounts for the largest proportion in the distribution of facial expression recognition results; Determine the distribution of learning behavior recognition results for each learner, and determine the learning behavior parameters of the target learning group based on the learning behavior category that accounts for the largest proportion in the distribution of learning behavior recognition results; The learning status detection result of the target learning group at the corresponding time is obtained by weighted calculation based on the facial expression parameters and the learning behavior parameters.

5. An apparatus for detecting learning state based on the cross-modal representation learning method according to any one of claims 1-4, characterized in that, The device includes: The data acquisition module is used to acquire the first video information of the target learning group captured by the RGB camera and the second video information of the target learning group captured by the dynamic vision camera; The image processing module is used to process the first video information to obtain a first image sequence and to process the second video information to obtain a second image sequence. The detection module is used to input the first image sequence and the second image sequence into a trained learning state detection network model, and to obtain the facial expression recognition result of each learner in the target learning group based on the first image sequence through the first sub-model of the learning state detection network model, and to obtain the learning behavior recognition result of each learner in the target learning group based on the second image sequence through the second sub-model of the learning state detection network model. The detection module is also used to determine the learning status detection result of each learner in the target learning group at a corresponding time based on the facial expression recognition result and the learning behavior recognition result.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 4.

Images