A student activity accompanying data analysis method, device and equipment

Abstract

The application discloses a kind of student activity accompanying type data analysis method, device and equipment, the method includes: obtaining the accompanying video data of object to be analyzed in learning activity process and carries out face detection, generates face image;Face feature point sampling is carried out to face image using a local face-based self-attention feature point sampling model, obtain the set of multiple sets of face feature points;The set of face feature points is input into a residual prompt-based self-attention feature extraction model to extract face feature;The face feature is compared with the face feature stored in face feature library, and face feature data and its corresponding identity information are generated;Based on the accompanying video data and face feature data, visual analysis is carried out.The application can improve the accuracy of face recognition, and based on the recognition result, the accompanying face data is visualized, so as to realize the demand of accurately guiding students to learn.

Description

Technical Field

[0001] This application relates to the field of facial data collection and recognition technology, and more specifically, to a method, apparatus and equipment for student activity-related data analysis. Background Technology

[0002] The development of artificial intelligence technology and the arrival of the big data era are accelerating personalized learning for students, facilitating schools to provide professional and individualized instruction and abandoning the traditional "assembly line" teaching model. Through data empowerment, schools can rely on data management platforms to collect accompanying data on student learning activities, and conduct dynamic analysis based on this data. Teachers and parents can then use this dynamic analysis to provide precise guidance for student learning, better serving student growth. However, due to the complexity of student learning activities and the unique nature of learning behavior, existing digital trajectories or digital profiles of student learning activities only scratch the surface, relying mainly on static and periodic data for indirect statistical analysis or presentation of student learning activities. There is an urgent need for accompanying data collection on student learning activities to achieve comprehensive and continuous data coverage and collection.

[0003] Currently, the collection and analysis of students' learning focus is mainly concentrated in schools, neglecting the collection and analysis of students' focus during learning activities at home. Furthermore, the prerequisite for using student learning data for visual dynamic analysis is accurate facial detection and recognition based on that data; however, current facial recognition technology has low accuracy in detecting blurry or occluded facial images, and the overall computational complexity is high. Summary of the Invention

[0004] To address at least one deficiency or improvement need in the existing technology, the present invention provides a method, apparatus, and device for analyzing accompanying student activity data. The purpose is to improve the accuracy of facial recognition and to perform visual analysis of accompanying facial data based on the recognition results, thereby achieving the need for precise guidance of student learning.

[0005] To achieve the above objectives, according to a first aspect of the present invention, a method for analyzing student activity-related data is provided, comprising the following steps:

[0006] S1 acquires accompanying video data of the object to be analyzed during the learning activity and performs face detection to generate face images;

[0007] S2 utilizes a self-attention feature point sampling model based on local faces to sample facial feature points in face images, obtaining a set of multiple sets of facial feature points; including:

[0008] Extract image features from facial images to generate feature maps;

[0009] A local image patch of a preset size is cropped from the feature map, centered on the predicted facial feature points;

[0010] The local coordinates of each facial feature point in the corresponding image block are extracted from the local image block. The sum of the local coordinates of the facial feature point and the relative position coordinates of the corresponding local image block is used as the final predicted coordinates to generate a set of facial feature points.

[0011] S3 inputs the set of facial feature points into a self-attention feature extraction model based on residual cues to extract facial features;

[0012] S4 compares the facial features with the facial features stored in the facial feature database to generate facial feature data and its corresponding identity information;

[0013] S5 performs visualization analysis based on the accompanying video data and facial feature data.

[0014] Furthermore, in the above-mentioned student activity-based data analysis method, the step of extracting the local coordinates of each facial feature point in the corresponding image patch from the local image patch includes:

[0015] Local image patches are mapped to vectors, and the distance between facial feature points in each local image patch is encoded to obtain the relative positional relationship of facial feature points in the local image patch;

[0016] Based on the encoding results, the relative positional relationship between facial feature points in each local image block is modeled, and the local coordinates of each facial feature point relative to the local image block are output.

[0017] Furthermore, in the aforementioned student activity-based data analysis method, the step of inputting a set of facial feature points into a self-attention feature extraction model based on residual cues to extract facial features includes:

[0018] The feature map is divided into image blocks centered on the coordinates of each facial feature point. Each image block is mapped to a two-dimensional vector and a classification label and position code are added.

[0019] By superimposing the image patches that have undergone position encoding, an aggregate vector is obtained;

[0020] The aggregated vector is input into the self-attention feature extraction model for feature extraction. The extraction process is as follows:

[0021]

[0022] Among them, the prompt symbol S is introduced. 0 ∈R M×d The prompt symbol S 0Along with image patches and classification labels, each image patch is transmitted through the encoder. At each layer, each image patch interacts with other image patches through a self-attention layer; `Split` represents splitting the propagation sequence of length N+L+1 into N+1 image patches and L cues; S J As the cue vector, for S J The final representation is obtained after performing average pooling. I0 J The classification label is the output value after passing through the Transformer encoder; x represents the final output of the entire model, using the loss function. The output x is used for supervised training in face recognition, where N is the number of samples and d is the margin. Among W is the weighting matrix of the last linear layer. i Let y represent the normalized i-th column of the weight matrix. j For the corresponding truth value, among them b j Let j be the j-th sample.

[0023] Furthermore, in the above-mentioned student activity-based data analysis method, the calculation of the l-th layer MSA during feature extraction can be expressed as:

[0024]

[0025] in, For learnable parameters, each self-attention s h Perform attention self-attention calculation using matrices Fusion of each self-attention head h Get R;

[0026] Residual hints will be provided during calculations within the MSA block of intermediate layer l. Added to the propagation hints, the calculation of the residual hints can be expressed as:

[0027]

[0028] in, It is a b×(N+1) dimensional zero matrix. For residual prompts; only add residual prompts to the propagation prompt locations, not to the locations of image patches or classification tags.

[0029] According to a second aspect of the present invention, a student activity-based data analysis device is also provided, comprising:

[0030] A video capture device is used to collect accompanying video data of the object to be analyzed during the learning process;

[0031] A face detector is used to detect faces in the accompanying video data and generate face images;

[0032] The face recognition module includes a self-attention feature point sampling model based on local faces and a self-attention feature extraction model based on residual cues;

[0033] The self-attention feature point sampling model is configured as follows:

[0034] Extract image features from facial images to generate feature maps;

[0035] A local image patch of a preset size is cropped from the feature map, centered on the predicted facial feature points;

[0036] The local coordinates of each facial feature point in the corresponding image block are extracted from the local image block. The sum of the local coordinates of the facial feature point and the relative position coordinates of the corresponding local image block is used as the final predicted coordinates to generate a set of facial feature points.

[0037] The self-attention feature extraction model is configured to extract features from the set of facial feature points to generate facial features;

[0038] The comparison module is used to compare the facial features with the facial features stored in the facial feature database to generate facial feature data and its corresponding identity information.

[0039] The analysis module performs visual analysis based on the accompanying video data and facial feature data.

[0040] Furthermore, in the aforementioned student activity-accompanied data analysis device, the self-attention feature point sampling model includes:

[0041] The backbone network is used to extract image features from face images and generate feature maps;

[0042] A linear embedding layer is used to crop a local image patch of a preset size from the feature map, centered on facial feature points, and map it into a vector;

[0043] The distance coding layer is used to encode the distance between facial feature points in each local image patch to obtain the relative positional relationship of facial feature points in the local image patch;

[0044] The relation modeling layer is used to model the relative positional relationships between facial feature points in each local image patch, and outputs the local coordinates of each facial feature point relative to the local image patch.

[0045] The coordinate prediction layer outputs the final predicted coordinates of the facial feature points, which are the sum of the local coordinates of the facial feature points and the relative position coordinates of the corresponding local image blocks.

[0046] Furthermore, in the aforementioned student activity-based data analysis device, the relationship modeling layer includes an MSA block, an MMA block, and an MLP block, with a normalization layer set before each block.

[0047] Furthermore, in the aforementioned student activity-based data analysis device, the self-attention feature extraction model includes:

[0048] The preprocessing layer is used to divide the image into blocks centered on the coordinates of each facial feature point on the feature map;

[0049] The location embedding layer maps each image patch to a two-dimensional vector and adds a classification label and location code. The image patches with location codes are superimposed to obtain an aggregate vector.

[0050] The feature extraction layer is used to extract facial features based on the aggregated vector. The extraction process is represented as follows:

[0051]

[0052] Among them, the prompt symbol S is introduced. 0 ∈R M×d The prompt symbol S 0 Along with image patches and classification labels, each image patch is transmitted through the encoder. At each layer, each image patch interacts with other image patches through a self-attention layer; `Split` represents splitting the propagation sequence of length N+L+1 into N+1 image patches and L cues; S J As the cue vector, for S J The final representation is obtained after performing average pooling. I0 J The classification label is the output value after passing through the Transformer encoder; x represents the final output of the entire model, using the loss function. The output x is used for supervised training in face recognition, where N is the number of samples and d is the margin. Among W is the weighting matrix of the last linear layer. i Let y represent the normalized i-th column of the weight matrix. j For the corresponding truth value, among them b j Let j be the j-th sample.

[0053] Furthermore, in the aforementioned student activity-associated data analysis device, the MSA calculation in the feature extraction layer of the l-th layer can be expressed as:

[0054]

[0055] in, For learnable parameters, each self-attention s h Perform attention self-attention calculation using matrices Fusion of each self-attention head h Get R;

[0056] Residual hints will be provided during calculations within the MSA block of intermediate layer l. Added to the propagation hints, the calculation of the residual hints can be expressed as:

[0057]

[0058] in, It is a b×(N+1) dimensional zero matrix. For residual prompts; only add residual prompts to the propagation prompt locations, not to the locations of image patches or classification tags.

[0059] According to a third aspect of the present invention, a student activity-based data analysis device is also provided, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program that, when executed by the processing unit, causes the processing unit to perform the steps of any of the methods described above.

[0060] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0061] (1) The face recognition model used in this invention utilizes a local face self-attention mechanism to learn the intrinsic relationships between facial feature points, improving the robustness of face alignment under blurred or heavily occluded conditions and reducing computational complexity. It also utilizes a residual-based self-attention mechanism to learn the features of face images with facial feature points. In various layered computations of the Transformer encoder, residual labels are added to the propagated cues to promote layered modulation without increasing the number of cues per layer. This expressive cue adjustment method with residual labels has the advantage of effective parameter adaptation and significantly improves performance in face recognition tasks.

[0062] (2) The present invention stores the accompanying facial data and video stream of student learning activities. Schools, teachers and parents can conduct dynamic analysis on this basis and use the dynamic analysis to accurately guide students' learning. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 A logic block diagram of a student activity-based data analysis device provided in this application embodiment;

[0065] Figure 2 A schematic diagram of the composition structure of the face detector provided in the embodiments of this application;

[0066] Figure 3 A schematic diagram of the network structure of the self-attention feature point sampling model provided in the embodiments of this application;

[0067] Figure 4 This is a schematic diagram illustrating the composition structure of the relationship modeling layer provided in an embodiment of this application;

[0068] Figure 5 A schematic diagram of the network structure of the self-attention feature extraction model provided in the embodiments of this application;

[0069] Figure 6 This is a flowchart illustrating a student activity-based data analysis method provided in an embodiment of this application. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0071] The terms "first," "second," "third," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0072] Furthermore, to avoid obscuring the understanding of the invention by those skilled in the art, well-known or widely used techniques, elements, structures, and processes may not be described or shown in detail. Although the accompanying drawings illustrate exemplary embodiments of the invention, the drawings are not necessarily drawn to scale, and specific features may be enlarged or omitted to better illustrate and explain the invention.

[0073] This embodiment provides a student activity-based data analysis device, which can be implemented in software and / or hardware and can be integrated into an electronic device; Figure 1 This is a schematic diagram of the analysis device provided in this embodiment. Please refer to it. Figure 1 The device includes a video capture unit, a face detector, a face recognition module, a comparison module, and an analysis module; among which,

[0074] Video capture devices are used to collect accompanying video data of the object to be analyzed during the learning process;

[0075] In a specific example, the video capture device uses a monocular visible light camera to place the face of the subject to be analyzed (such as students and parents) within the detection area of ​​the monocular visible light camera and capture accompanying video data during their learning activities.

[0076] A face detector is used to detect faces in accompanying video data and generate face images;

[0077] Figure 2 The structure of the face detector is illustrated. Specifically, the face detector first reads the input accompanying video data and performs adaptive image scaling on each frame. The data is then input into the backbone network for feature extraction. The extracted features are then input into a feature fusion module for multi-scale feature fusion. The prediction head module predicts and regresses targets based on the fused features. The predicted bounding boxes output by the prediction head module are converted into a 4D format through normalization and convolution operations. The classification prediction branch is processed using the sigmoid activation function, and the bounding box prediction branch is decoded. The three different-scale classification prediction branches and bounding box branches output by the prediction head module are concatenated, and non-maximum suppression (NMS) is performed to generate the detected face image X. l ∈R H×W×C (C=3), l=1,...,L-1; L represents the number of face images.

[0078] In this embodiment, a multi-head pooling self-attention module (PSAE) and a lightweight convolutional attention module (CBAM) are introduced into the network structure of the face detector. The spatial and channel attention mechanisms of CBAM are used to extract the attention regions of the image, making the network model more focused on the target object. The processing flow of the PSAE module can be represented as follows:

[0079]

[0080] In the above formula, LN refers to Layer Normalization, which aims to normalize the entire image. Polarimetric Self-Attention (PSA) is an improvement on Multi-Head Self-Attention (MSA) to reduce memory usage. A Multilayer Perceptron (MLP) consists of fully connected layers, a GELU activation function, Depthwise convolution, and another fully connected layer. The calculation of self-attention in MSA can be expressed as:

[0081]

[0082] in, These are matrices consisting of the query vector, key vector, and value vector, respectively, where n is the length of the sequence and d is the value of the key vector. h Let be the dimension of the vector.

[0083] The calculation of each self-attention point in PSA can be expressed as:

[0084]

[0085] In this case, the dimension of U remains unchanged, while the dimension of E and Y matrices becomes P after average pooling. 2 ×C i P is the pooling size in PSA, and C is the pooling size in PSA. i The number of output channels in stage i. d h Let be the dimension of the vector. SR(E) and SR(Y) refer to downsampling matrices E and Y through pooling operations, thereby reducing the complexity of multi-head self-attention to linear complexity.

[0086] Like MSA, PSA contains multiple self-attention heads. PSA concatenates the results from each self-attention head and then outputs the concatenated result via... The fusion can be represented as:

[0087]

[0088] The face recognition module includes a self-attention feature point sampling model based on local faces and a self-attention feature extraction model based on residual cues;

[0089] Figure 3 This is a schematic diagram of the network structure of the self-attention feature point sampling model provided in this embodiment, as shown below. Figure 3 As shown, the self-attention feature point sampling model includes a backbone network, a linear embedding layer, a distance encoding layer, a relation modeling layer, and a coordinate prediction layer;

[0090] The backbone network is used to extract image features from face images and generate feature maps;

[0091] In a specific example, the backbone network uses the HRNet network; the HRNet network inputs the extracted feature maps into the linear embedding layer for processing;

[0092] The linear embedding layer is used to crop local image patches of a preset size from the feature map with facial feature points as the center. Each image patch is embedded into a vector, which can be regarded as the representation of the corresponding facial feature points.

[0093] The distance coding layer encodes the distances between facial feature points in each local image patch to obtain the relative positional relationships of facial feature points in the local image patch.

[0094] The relation modeling layer models the relative positional relationships between facial feature points in each local image patch, and outputs the local coordinates of each facial feature point relative to the local image patch.

[0095] The coordinate prediction layer outputs the final predicted coordinates f of the facial feature points. i =[x i ,y i ], i = 1, ... S 2 These coordinates are the local coordinates of facial feature points. The sum of the coordinates of the corresponding local image patch; the facial feature point f i The set F is used as the input to the self-attention feature extraction model.

[0096] Figure 4 This is a schematic diagram of the composition structure of the relational modeling layer provided in this embodiment. In an optional implementation, the relational modeling layer mainly includes an MSA block, an MMA block, and an MLP block, and Layer Norm is applied before each block for layer normalization. The query vector L for each facial feature point is learned using the MSA block. q The relationship between the facial feature points and the query vector L is learned using MMA blocks. q The relationship between them. Then the query vector L of facial feature points. q The input is fed into the decoder to learn the inherent relationships between each feature point, and the output is fed back into the shared MLP to predict the location of each facial feature point. Finally, the predicted facial feature point f is output. i The set F.

[0097] MSA block computation can be represented as:

[0098]

[0099] in, For the learnable parameters of the linear layer, Let L represent the input of the i-th layer self-attention head and L respectively. q , C I For dimensions.

[0100] The MMA block utilizes a cross-self-attention mechanism to learn the point feature representation of feature points and the query vector L. q The process of the relationship is as follows:

[0101]

[0102] in, For the learnable parameters of the linear layer, For the input of the i-th layer MMA block, For position encoding, This represents the positional characteristics of the feature points.

[0103] like Figure 4 As shown, the MLP block contains a fully connected layer, a GELU activation function, a DW convolution (Depthwise convolution), and a fully connected layer.

[0104] The self-attention feature extraction model is configured to extract features from a set of facial feature points to generate facial features; Figure 5 This is a schematic diagram of the network structure of the self-attention feature extraction model provided in this embodiment. As an optional implementation, the self-attention feature extraction model includes a preprocessing layer, a location embedding layer, and a feature extraction layer.

[0105] The preprocessing layer is used to divide the image into blocks on the feature map centered on the coordinates of each facial feature point; in a specific example, the STN differentiable grid sampling method is used to extract each image block.

[0106] The location embedding layer is used to map each image patch to a two-dimensional vector and add classification labels and location codes. The image patches with location codes are superimposed to obtain an aggregate vector.

[0107] In this embodiment, a linear embedding layer is used. Each image patch is mapped to a two-dimensional vector, denoted as Token. A classification token is added before these Tokens, and positional encoding is added to these Tokens. Finally, these positionally encoded Tokens are superimposed to obtain the aggregate vector I. 0 The process of location embedding is as follows:

[0108]

[0109] Wherein, vector I 0 For tokens with overlaid position codes, M pos ∈R (N+1)×D .

[0110] The feature extraction layer is used to extract facial features based on the aggregate vector. The extraction process is represented as follows:

[0111]

[0112] Among them, the prompt symbol S is introduced. 0 ∈R M×d The prompt symbol S 0 Along with image patches and classification labels, each image patch is transmitted through the encoder. At each layer, each image patch interacts with other image patches through a self-attention layer; `Split` represents splitting the propagation sequence of length N+L+1 into N+1 image patches and L cues; S J As the cue vector, for S J The final representation is obtained after performing average pooling. I0 J The classification label is the output value after passing through the Transformer encoder; x represents the final output of the entire model, using the loss function. The output x is used for supervised training in face recognition, where N is the number of samples and d is the margin. Among W is the weighting matrix of the last linear layer. i Let y represent the normalized i-th column of the weight matrix. j For the corresponding truth value, among them b j Let j be the j-th sample.

[0113] In the feature extraction layer, the MSA calculation of the l-th layer can be expressed as:

[0114]

[0115] in, For learnable parameters, each self-attention s h Perform attention self-attention calculation using matrices Fusion of each self-attention head h We get R.

[0116] Shallow cues can model some of the necessary token relationships for the task, while residual cues help to independently reweight the attention weights for a specific task at each layer, aggregating and adjusting the contextual information of image patch tokens. To enhance cues, hierarchical residual cues are introduced. In this embodiment, residual cues are incorporated into the computation within the MSA block of the intermediate layer 1. Added to the propagation hints, the calculation of the residual hints can be expressed as:

[0117]

[0118] in, It is a b×(N+1) dimensional zero matrix. This is the residual indicator.

[0119] In this embodiment, residual cues are added only to the propagated cue positions, not to the positions of image patches or classification tags. In the various hierarchical computations of the Transformer encoder, residual tags are added to the propagated cues to facilitate hierarchical modulation without increasing the number of cues per layer. This expressive cue adjustment method with residual tags offers the advantage of efficient parameter adaptation while significantly improving performance in face recognition tasks.

[0120] The comparison module is used to compare facial features with facial features stored in the facial feature database to generate facial feature data and its corresponding identity information;

[0121] Specifically, the comparison module compares the extracted facial features with the facial feature database of students and their parents recorded in the facial feature database to obtain the facial recognition result. If the recognition is successful, the identity information of the student and their parents is output; if the recognition fails, the recognition failure is reported.

[0122] The analysis module performs visual analysis based on accompanying video data and facial feature data.

[0123] In this embodiment, the system records the detected facial images and facial recognition information every minute during the student's learning activities. When the student's learning activities end, the video stream of the learning activities and the accompanying facial data are stored in the student's accompanying database. Based on this stored data, visualization analysis is performed. As an optional implementation, the analysis module counts the times when no face is detected and when face recognition fails to identify the student. These times represent when the student's face leaves the learning area. A pie chart is used to visualize the frequency of student faces leaving the learning area, which can indirectly reflect the student's level of focus. Furthermore, the system counts the times when parents appear in the student's learning area and uses a pie chart to visualize the frequency of parents' faces appearing in the student's learning area. The frequency of parents' faces appearing in the student's learning area can indirectly reflect the degree of parental involvement in the student's learning activities. Based on the location and duration of student faces appearing, a heatmap is generated. A coordinate system is established with the center position of the student's face at the start of the learning activity as the origin. The system then counts the positions the student's face moves to and the duration of time the face stays at each position during the learning process. Each time a student moves to a different location, a point appears on the heatmap. The longer the student stays in a particular position, the darker the red of that point becomes, and vice versa. The generated heatmap can reveal changes in the position of students' faces during learning, and can be used to analyze students' concentration levels during learning activities.

[0124] Figure 6 This is a flowchart illustrating a student activity-based data analysis method provided in this embodiment. Please refer to [link / reference]. Figure 6 The method mainly includes the following steps:

[0125] S1 acquires accompanying video data of the object to be analyzed during the learning activity and performs face detection to generate face images;

[0126] In this step, the subjects to be analyzed are students who complete learning activities at home, and may further include parents who participate in the students' learning process.

[0127] In a specific example, a visible light camera is used to collect accompanying video data of students during their learning activities. During data collection, it must be ensured that the faces of students and parents are within the detectable area of ​​the monocular visible light camera. This accompanying video data is then input into a face detector for face detection. If no face image is detected, the time of failure is saved to a cloud database; if a face image is detected, the detected face image X is output. l .

[0128] S2 utilizes a self-attention feature point sampling model based on local faces to sample facial feature points in face images, obtaining a set of multiple sets of facial feature points; including:

[0129] S21 extracts image features from the face image and generates a feature map;

[0130] In a specific example, a face image is input into the HRNet network for feature extraction to form a feature map.

[0131] S22 uses the predicted facial feature point coordinates as the center to crop a local image patch of a preset size from the feature map;

[0132] In this embodiment, the prediction of facial feature points is divided into multiple stages. As the number of stages increases, the predicted feature point positions become more accurate. The first stage uses the coordinates of a set of standard frontal facial feature points of P = S × S as prior knowledge. In each subsequent stage, a local image block of a preset size is cropped from the feature map centered on the facial feature points of the previous stage.

[0133] S23 extracts the local coordinates of each facial feature point in the corresponding image block from the local image block, and uses the sum of the local coordinates of the facial feature point and the relative position coordinates of the corresponding local image block as the final predicted coordinates to generate a set of facial feature points.

[0134] As an optional implementation, extracting the local coordinates of each facial feature point in the corresponding image patch from the local image patch includes:

[0135] Local image patches are mapped to vectors, and the distance between facial feature points in each local image patch is encoded to obtain the relative positional relationship of facial feature points in the local image patch;

[0136] Based on the encoding results, the relative positional relationship between facial feature points in each local image block is modeled, and the local coordinates of each facial feature point relative to the local image block are output.

[0137] S3 inputs the set of facial feature points into a self-attention feature extraction model based on residual cues to extract facial features;

[0138] As an optional implementation, the facial feature extraction process includes:

[0139] S31 divides the image into blocks centered on the coordinates of each facial feature point on the feature map;

[0140] S32 maps each image patch to a two-dimensional vector and adds a classification label and location code. The image patches with location codes are superimposed to obtain an aggregate vector.

[0141] Specifically, using linear embedding layers Each image patch is mapped to a two-dimensional vector, denoted as Token. A classification token is added before these Tokens, and positional encoding is added to these Tokens. Finally, these positionally encoded Tokens are superimposed to obtain vector I. 0 The process of position embedding is represented as:

[0142]

[0143] Wherein, vector I 0 For tokens with overlaid position codes, M pos ∈R (N+1)×D .

[0144] S33 inputs the aggregated vector into the self-attention feature extraction model for feature extraction. The extraction process is represented as follows:

[0145]

[0146] Among them, the prompt symbol S is introduced. 0 ∈R M×d The prompt symbol S 0 Along with image patches and classification labels, each image patch is transmitted through the encoder. At each layer, each image patch interacts with other image patches through a self-attention layer; `Split` represents splitting the propagation sequence of length N+L+1 into N+1 image patches and L cues; S J As the cue vector, for S J The final representation is obtained after performing average pooling. I0 J The classification label is the output value after passing through the Transformer encoder; x represents the final output of the entire model, using the loss function. The output x is used for supervised training in face recognition, where N is the number of samples and d is the margin. Among W is the weighting matrix of the last linear layer. i Let y represent the normalized i-th column of the weight matrix. j For the corresponding truth value, among them b j Let j be the j-th sample.

[0147] In the feature extraction process, the calculation of the l-th layer MSA can be expressed as:

[0148]

[0149] in, For learnable parameters, each self-attention s h Perform attention self-attention calculation using matrices Fusion of each self-attention head h Get R;

[0150] Residual hints will be provided during calculations within the MSA block of intermediate layer l. Added to the propagation hints, the calculation of the residual hints can be expressed as:

[0151]

[0152] in, It is a b×(N+1) dimensional zero matrix. This is the residual indicator.

[0153] In this embodiment, residual hints are added only to the propagation hint positions, and not to the positions of image patches or classification tags.

[0154] S4 compares facial features with facial features stored in the facial feature database to generate facial feature data and its corresponding identity information;

[0155] S5 performs visualization analysis based on accompanying video data and facial feature data.

[0156] For specific details regarding the limitations of the accompanying data analysis method for student activities, please refer to the limitations of the accompanying data analysis device for student activities mentioned above. Repeated details will not be repeated here.

[0157] It should be noted that although the operations of the methods of the embodiments of this specification are described in a specific order in the above embodiments, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowcharts may be executed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0158] This embodiment also provides a student activity-based data analysis device, which includes at least one processor and at least one memory. The memory stores a computer program. When the computer program is executed by the processor, the processor performs the steps of the student activity-based data analysis method. The specific steps are described above and will not be repeated here. In this embodiment, the types of processor and memory are not specifically limited. For example, the processor can be a microprocessor, a digital information processor, an on-chip programmable logic system, etc.; the memory can be volatile memory, non-volatile memory, or a combination thereof.

[0159] The electronic device can also communicate with one or more external devices (such as a keyboard, pointing terminal, display, etc.), one or more terminals that enable users to interact with the electronic device, and / or any terminal that enables the electronic device to communicate with one or more other computing terminals (such as a network card, modem, etc.). This communication can be performed via an input / output (I / O) interface. Furthermore, the electronic device can also communicate with one or more networks (such as a Local Area Network (LAN), a Wide Area Network (WAN), and / or a public network, such as the Internet) via a network adapter.

[0160] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0161] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0162] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0163] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0164] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for analyzing data accompanying student activities, characterized in that, Includes the following steps: S1 acquires accompanying video data of the object to be analyzed during the learning activity and performs face detection to generate face images; S2 utilizes a self-attention feature point sampling model based on local faces to sample facial feature points in face images, obtaining a set of multiple sets of facial feature points; including: Extract image features from facial images to generate feature maps; A local image patch of a preset size is cropped from the feature map, centered on the predicted facial feature points; The local coordinates of each facial feature point in the corresponding image block are extracted from the local image block. The sum of the local coordinates of the facial feature point and the relative position coordinates of the corresponding local image block is used as the final predicted coordinates to generate a set of facial feature points. S3 inputs the set of facial feature points into a self-attention feature extraction model based on residual cues to extract facial features, including: The feature map is divided into image blocks centered on the coordinates of each facial feature point. Each image block is mapped to a two-dimensional vector and a classification label and position code are added. By superimposing the image patches that have undergone position encoding, an aggregate vector is obtained; The aggregated vector is input into the self-attention feature extraction model for feature extraction. The extraction process is as follows: Among them, prompt symbols are introduced. The prompt symbol Along with the image patches and classification labels, they are transmitted through the encoder. At each layer, each image patch interacts with other image patches through a self-attention layer; Split means splitting the propagation sequence of length N+L+1 into N+1 image patches and L cues. For the cue vector, The final representation is obtained after performing average pooling. ; The output value of the classification label after passing through the Transformer encoder; x The final output of the entire model is represented by the loss function. For output x Supervised training for face recognition is performed, where N is the number of samples; For margin, Among them It is the weighting matrix of the last linear layer. The normalized first weight matrix represents the weight matrix. i List, For the corresponding truth value, among them , For the first j One sample; S4 compares the facial features with the facial features stored in the facial feature database to generate facial feature data and its corresponding identity information; S5 performs visualization analysis based on the accompanying video data and facial feature data.

2. The student activity-based data analysis method as described in claim 1, characterized in that, The step of extracting the local coordinates of each facial feature point in the corresponding image patch from the local image patch includes: Local image patches are mapped to vectors, and the distance between facial feature points in each local image patch is encoded to obtain the relative positional relationship of facial feature points in the local image patch; Based on the encoding results, the relative positional relationship between facial feature points in each local image block is modeled, and the local coordinates of each facial feature point relative to the local image block are output.

3. The student activity-based data analysis method as described in claim 1, characterized in that, In the feature extraction process, the first Layer MSA calculation can be expressed as: in, , , , For learnable parameters, each self-attention... Perform self-attention calculation using matrices Integrate each self-focused head Get R; In the middle layer The residuals will be indicated in the calculation within the MSA block. Added to the propagation hints, the calculation of the residual hints can be expressed as: in, It is a b×(N+1) dimensional zero matrix. For residual prompts; only add residual prompts to the propagation prompt locations, not to the locations of image patches or classification tags.

4. A student activity-based data analysis device, characterized in that, include: A video capture device is used to collect accompanying video data of the object to be analyzed during the learning process; A face detector is used to detect faces in the accompanying video data and generate face images; The face recognition module includes a self-attention feature point sampling model based on local faces and a self-attention feature extraction model based on residual cues; The self-attention feature point sampling model is configured as follows: Extract image features from facial images to generate feature maps; A local image patch of a preset size is cropped from the feature map, centered on the predicted facial feature points; The local coordinates of each facial feature point in the corresponding image block are extracted from the local image block. The sum of the local coordinates of the facial feature point and the relative position coordinates of the corresponding local image block is used as the final predicted coordinates to generate a set of facial feature points. The self-attention feature extraction model is configured to extract features from the set of facial feature points to generate facial features; the self-attention feature extraction model includes: The preprocessing layer is used to divide the image into blocks centered on the coordinates of each facial feature point on the feature map; The location embedding layer maps each image patch to a two-dimensional vector and adds a classification label and location code. The image patches with location codes are superimposed to obtain an aggregate vector. The feature extraction layer is used to extract facial features based on the aggregated vector. The extraction process is represented as follows: Among them, prompt symbols are introduced. The prompt symbol Along with the image patches and classification labels, they are transmitted through the encoder. At each layer, each image patch interacts with other image patches through a self-attention layer; Split means splitting the propagation sequence of length N+L+1 into N+1 image patches and L cues. For the cue vector, The final representation is obtained after performing average pooling. ; The output value of the classification label after passing through the Transformer encoder; x The final output of the entire model is represented by the loss function. For output x Supervised training for face recognition is performed, where N is the number of samples; For margin, Among them It is the weighting matrix of the last linear layer. The normalized first weight matrix represents the weight matrix. i List, For the corresponding truth value, among them , For the first j One sample; The comparison module is used to compare the facial features with the facial features stored in the facial feature database to generate facial feature data and its corresponding identity information. The analysis module performs visual analysis based on the accompanying video data and facial feature data.

5. The student activity-based data analysis device as described in claim 4, characterized in that, The self-attention feature point sampling model includes: The backbone network is used to extract image features from face images and generate feature maps; A linear embedding layer is used to crop a local image patch of a preset size from the feature map, centered on facial feature points, and map it into a vector; The distance coding layer is used to encode the distance between facial feature points in each local image patch to obtain the relative positional relationship of facial feature points in the local image patch; The relationship modeling layer is used to model the relative positional relationships between facial feature points in each local image patch, and outputs the local coordinates of each facial feature point relative to the local image patch; The coordinate prediction layer outputs the final predicted coordinates of the facial feature points, which are the sum of the local coordinates of the facial feature points and the relative position coordinates of the corresponding local image blocks.

6. The student activity-based data analysis device as described in claim 5, characterized in that, The relation modeling layer includes an MSA block, an MMA block, and an MLP block, with a normalization layer set before each block.

7. The student activity-based data analysis device as described in claim 4, characterized in that, In the feature extraction layer, the first Layer MSA calculation can be expressed as: in, , , , For learnable parameters, each self-attention... Perform self-attention calculation using matrices Integrate each self-focused head Get R; In the middle layer The residuals will be indicated in the calculation within the MSA block. Added to the propagation hints, the calculation of the residual hints can be expressed as: in, It is a b×(N+1) dimensional zero matrix. For residual prompts; only add residual prompts to the propagation prompt locations, not to the locations of image patches or classification tags.

8. A student activity-based data analysis device, characterized in that, It includes at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program that, when executed by the processing unit, causes the processing unit to perform the steps of the method according to any one of claims 1 to 3.

Images