A method, device, terminal device and medium for identifying student spatiotemporal movements

Abstract

The present application is applicable to the technical field of teaching management systems, and provides a method, apparatus, terminal equipment and medium for identifying spatiotemporal actions of students. The method comprises: constructing a face feature extraction model based on an improved face recognition method and an image feature extraction neural network model; constructing a spatiotemporal action recognition model based on a human body detection model and a spatiotemporal feature extraction neural network model; processing the video data to be processed using the face detection model and the spatiotemporal action recognition model to obtain a set of action tubes and a set of face annotation frames; spatially intersecting the set of action tubes and the set of face annotation frames to obtain a set of actions to be identified; extracting facial feature data of each face annotation frame using the face feature extraction model; matching the facial feature data of the target student with the pre-processed facial feature data to obtain the identity information and action information of the target student. The present application can match the spatiotemporal actions of students with their identities.

Description

Technical Field

[0001] The present application belongs to the technical field of teaching management systems, and in particular relates to a method, apparatus, terminal device and medium for identifying spatiotemporal movements of students. Background Art

[0002] With the advancement of the national education informatization plan, an increasing number of artificial intelligence technologies and intelligent edge devices are being applied to high-quality information-based teaching practices. This has generated a large amount of raw data for research in teaching scenarios. This data is multimodal and contains various aspects of student information. By using artificial intelligence technology to mine and analyze this video, audio, and text data, the information required for the target task can be obtained. In particular, student observation video data in classroom teaching scenarios has the characteristics of large capacity and easy access. Therefore, data mining of this data has become a hot topic in the current intersection of computer vision and education. Current research on student observation video data focuses on two general research directions: motion recognition and face recognition. By recognizing students' movements and faces in the classroom, applications such as classroom attendance and teaching quality verification can be realized.

[0003] Facial recognition methods typically involve two stages: face detection and face verification. In particular, current facial verification methods for classroom scenarios mostly perform identity matching on a small scale and are not generalizable to large-scale student identity verification scenarios. Current motion recognition methods, on the other hand, focus solely on identifying classroom movements. These movements are not mapped to specific student identities, and the acquired motion data can only be used for overall teaching evaluation within a specific classroom, failing to analyze the movements of specific students.

[0004] In summary, current methods for identifying students’ spatiotemporal motions are unable to match students’ spatiotemporal motions with their identities. Summary of the Invention

[0005] The embodiments of the present application provide a method, apparatus, terminal device, and medium for identifying spatiotemporal movements of students, which can solve the problem that the spatiotemporal movement identification method of students cannot match the spatiotemporal movements of students with their identities.

[0006] In a first aspect, an embodiment of the present application provides a method for identifying spatiotemporal movements of students, comprising:

[0007] Construct a facial feature extraction model based on the improved face recognition method and image feature extraction neural network model;

[0008] Construct a spatiotemporal action recognition model based on the pre-trained human detection model and spatiotemporal feature extraction neural network model;

[0009] The pre-trained face detection model and spatiotemporal action recognition model are used to process the video data to obtain an action tube set and a face annotation frame set; the action tube set includes multiple action annotation frame sets;

[0010] Performing a spatial intersection operation on the action tube set and the face annotation box set to obtain a set of actions to be recognized; the set of actions to be recognized includes multiple face annotation boxes and an action annotation box corresponding to each face annotation box;

[0011] Extracting facial feature data of each face annotation frame in the set of actions to be identified using a facial feature extraction model; the facial feature data includes multiple facial features;

[0012] The facial feature data of the target student is matched with the pre-processed facial feature data in the feature database to obtain the identity information of the target student and the action information of the target student.

[0013] Optionally, a facial feature extraction model is constructed based on the improved facial recognition model and the image feature extraction neural network model, including:

[0014] By calculating the formula

[0015]

[0016] The improved ArcFace loss function L is obtained; where N represents the size of each training batch, n represents the number of categories, m1, m2, and m3 represent the multiplicative angle margin penalty of SphereFace, the additive angle margin penalty of ArcFace, and the additive cosine margin penalty of CosFace, respectively. θ represents the angle between the feature vector of the training batch sample and the column vector of the weight matrix. The subscript j represents the number of columns in the weight matrix. i represents the number of columns corresponding to the actual category of the feature vector of the i-th sample in the training batch, and s represents the radius of the hyperplane obtained by training;

[0017] The pre-trained PyramidVision Transformer v2-B2-li neural network model is used as the image feature extraction neural network model, and the improved ArcFace loss function L is used to optimize the Pyramid Vision Transformerv2-B2-li neural network model to obtain the face recognition feature extraction model.

[0018] Optionally, a spatiotemporal action recognition model is constructed based on a pre-trained human body detection model and a spatiotemporal feature extraction neural network model, including:

[0019] Use the pre-trained human detection model to detect the original video data and obtain multiple human annotation frames;

[0020] ResNet50 is used as the backbone network of the SlowFast neural network model, and the SlowFast neural network model is used to extract the spatiotemporal features of the original video data and the motion information of each human body annotation frame in multiple human body annotation frames;

[0021] A spatiotemporal action recognition model is constructed based on spatiotemporal features and action information.

[0022] Optionally, a spatial intersection operation is performed on the action tube set and the face annotation box set, including:

[0023] Separately align the action tube set A and the face annotation frame set F with the video data to be processed to obtain the action tube set with timestamps and the face annotation frame set with timestamps; where A=A1,A2,...,A q ,F=F1,F2,...,F m , q represents the total number of action tubes, m represents the total number of face annotation boxes;

[0024] Double-cropping the action tube set with the timestamp to obtain a video frame set; the video frame set includes multiple action tubes with the timestamp;

[0025] For each action annotation box in the video frame set, the formula is calculated

[0026]

[0027] h inter =max(0,min(ay2,by2)-max(ay1,by1))

[0028] w inter =max(0,min(ax2,bx2)-max(ax1,bx1))

[0029] Get the intersection over union (IoU) of the action annotation box and all face annotation boxes at the same timestamp; where the coordinates of the action annotation box are (ax1, ay1, ax2, ay2), and the coordinates of the face annotation box are (bx1, by1, bx2, by2), IoU = IoU1, IoU2, ..., IoU g , g represents the total number of face annotation boxes corresponding to the action annotation box at the same timestamp;

[0030] The face annotation frame with the highest intersection-with-union ratio and the closest geometric distance to the center point of the action annotation frame is taken as the initiator of the action in the action annotation frame, and a set of face action data is obtained;

[0031] A set of actions to be recognized is constructed based on multiple sets of facial action data corresponding to all action annotation boxes.

[0032] Optionally, double-crop the action tube set with timestamps to obtain a video frame set, including:

[0033] Crop the action tube set containing the timestamp to obtain a first video slice;

[0034] Dividing the first video slice into regions using a normal distribution to obtain a plurality of video slice regions;

[0035] Video frames are randomly sampled from multiple video slice regions to obtain a video frame set.

[0036] Optionally, matching the target student's facial feature data with pre-processed facial feature data in a feature database includes:

[0037] By calculating the formula

[0038]

[0039] The cosine similarity cosθ between the target student’s facial features and the pre-processed facial features in the feature database is obtained, where the target student’s facial features are ax i Represents the i-th feature component of the target student’s facial features. The pre-processed facial features in the feature database are bx i represents the i-th feature component of the preprocessed facial features in the feature database, and n represents the total number of feature components of the student’s facial features;

[0040] If cosθ is greater than the preset threshold of cosine similarity, it is determined that the target student's facial features match the pre-processed facial features, and the identity information corresponding to the pre-processed facial features in the feature database is determined as the identity information of the target student;

[0041] The action information in the action annotation box corresponding to the facial feature data of the target student in the action set to be identified is determined as the action information of the target student.

[0042] Optionally, after constructing a facial feature extraction model based on the improved facial recognition model and the image feature extraction neural network model, the student spatiotemporal action recognition method provided by this application further includes:

[0043] The facial feature extraction model is used to extract the facial feature data of all students, and a feature database is constructed based on the facial feature data of all students.

[0044] In a second aspect, an embodiment of the present application provides a student spatiotemporal motion recognition device, comprising:

[0045] A facial feature module is used to construct a facial feature extraction model based on the improved face recognition method and the image feature extraction neural network model;

[0046] The spatiotemporal action recognition module is used to build a spatiotemporal action recognition model based on the pre-trained human body detection model and spatiotemporal feature extraction neural network model;

[0047] A video processing module is used to process the video data to be processed using a pre-trained face detection model and a spatiotemporal action recognition model to obtain an action tube set and a face annotation frame set; the action tube set includes multiple action annotation frame sets;

[0048] A spatial intersection module is used to perform a spatial intersection operation on the action tube set and the face annotation box set to obtain a set of actions to be recognized; the set of actions to be recognized includes multiple face annotation boxes and the action annotation box corresponding to each face annotation box;

[0049] A facial feature extraction module is used to extract facial feature data of each face annotation frame in the set of actions to be identified using a facial feature extraction model; the facial feature data includes multiple facial features;

[0050] The facial feature matching module is used to match the facial feature data of the target student with the pre-processed facial feature data in the feature database to obtain the identity information of the target student and the action information of the target student.

[0051] In a third aspect, an embodiment of the present application provides a terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned student spatiotemporal motion recognition method is implemented.

[0052] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned student spatiotemporal motion recognition method.

[0053] The above solution of the present application has the following beneficial effects:

[0054] In some embodiments of the present application, by performing a spatial intersection operation on the action tube set and the face annotation frame set, the spatiotemporal actions of the students in the action tube set can be matched with the face annotation frames; the facial feature data of the target student is matched with the pre-processed facial feature data in the feature database to obtain the identity information of the target student and the action information of the target student, so that the spatiotemporal actions of the students are matched with the identity of the students.

[0055] Other beneficial effects of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0057] Figure 1 A flow chart of a method for identifying spatiotemporal movements of students provided in one embodiment of the present application;

[0058] Figure 2 A schematic diagram of the structure of a device for identifying spatiotemporal movements of students provided in one embodiment of the present application;

[0059] Figure 3 A flow chart of processing a video to be processed by the student spatiotemporal action recognition method provided in one embodiment of the present application;

[0060] Figure 4 A schematic diagram of the structure of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0061] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0062] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0063] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0064] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0065] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0066] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0067] In response to the problem that current methods for identifying students' spatiotemporal motion are unable to match students' spatiotemporal motion with their identities, the present application provides a method for identifying students' spatiotemporal motion. By performing spatial intersection operations on the action tube set and the face annotation frame set, the spatiotemporal motion of students in the action tube set can be matched with the face annotation frame; the facial feature data of the target student is matched with the pre-processed facial feature data in the feature database to obtain the identity information of the target student and the motion information of the target student, thereby matching the student's spatiotemporal motion with the student's identity.

[0068] like Figure 1 As shown, the student spatiotemporal action recognition method provided by this application includes the following steps:

[0069] Step 11: construct a face feature extraction model based on the improved face recognition method and the image feature extraction neural network model.

[0070] After constructing the above-mentioned facial feature extraction model, it is also necessary to use the facial feature extraction model to extract the facial feature data of all students and build a facial feature database for subsequent matching of students' identities.

[0071] In some embodiments of the present application, when pre-training the image feature extraction neural network model, the ArcFace (a loss function (an operation function used to measure the degree of difference between the model's predicted value f(x) and the true value Y, which is a non-negative real-valued function, usually expressed as L(Y, f(x))), the smaller the loss function, the higher the z accuracy of the model)) margin penalty is improved, and the two margin penalty modes of SphereFace and CosFace (multiplicative angle margin penalty and additive cosine margin penalty) are added and integrated to enhance the feature space constructed by the network, further optimize the geodesic distance margin based on ArcFace, and make it more adaptable to large-scale feature extraction work.

[0072] In some embodiments of the present application, Pyramid Vision Transformer v2-B2-li (PVT-v2-B2-li) is used as the backbone network of the face feature extraction model (a neural network used to extract features from images) to process input images of different sizes. Compared with ResNet-50, the advantages of PVT-v2-B2-li are: PVT-v2-B2-li has fewer parameters at similar model sizes, consumes less computing resources, and has stronger image classification capabilities.

[0073] PVT-v2 extracts multi-scale features by stacking multiple Transformer Encoders (the encoder in the Transformer neural network can convert the input code into a machine-learnable vector. The encoder structure in the Pyramid Vision Transformer v2-B2-li used in this application consists of multiple encoder layers, each of which contains a linear spatial reduction attention layer (Li-SRA) and a feedforward layer) and simplifies the calculation by using the linear spatial reduction attention layer. The Li-SRA in stage i can be described by the following formula:

[0074]

[0075]

[0076] Among them, Concat(·) is the connection operation in attention calculation, is the parameter of the linear projection, C i is the number of feature channels at stage i, represents the real number field, N i is the number of heads in the multi-head attention layer in stage i, d headis the dimension of each attention head, Q represents the query vector, K represents the key vector, and V represents the value vector, which are obtained by multiplying the input with the corresponding matrix. ap (·) is an operation that reduces the dimension of the sequence space by using average pooling (average pooling is used to average all values ​​in the local receptive field, which is used in this application to reduce the size of the encoder output vector to reduce computational overhead). The formula is as follows:

[0077] SR ap (x)=Norm(AvgPool(x,S i )),

[0078] in represents the input sequence, S i represents the size of the attention layer in stage i, AvgPool(x,S i ) indicates that the input sequence x is processed with size S i Norm(·) represents the layer normalization operation (which performs layer normalization on the input features to reduce the fluctuation of the network layer).

[0079] Step 12: construct a spatiotemporal action recognition model based on the pre-trained human body detection model and spatiotemporal feature extraction neural network model.

[0080] The above-mentioned spatio-temporal action detection refers to simultaneously identifying the time range and corresponding category of the action, and marking the spatial position of the action initiator in the spatial range.

[0081] Step 13: Use the pre-trained face detection model and spatiotemporal action recognition model to process the video data to obtain an action tube set and a face annotation box set.

[0082] The aforementioned human and face detection models are essentially object detection models, used to determine the location and category of objects in an input image. Currently, there are two main types: one-stage and two-stage. One-stage models directly extract features through the network and predict the classification and location of the target, with the Yolo series being a typical example. Two-stage models first generate candidate regions and then predict the classification and location of the target through a neural network, with Faster-RCNN being a typical example. The aforementioned action tube represents an ordered collection of action bounding boxes organized in chronological order.

[0083] Step 14: perform a spatial intersection operation on the action tube set and the face annotation box set to obtain a set of actions to be recognized.

[0084] The spatial intersection operation is performed on the action tube set and the face annotation box set in order to match the spatiotemporal actions of the students in the action tube set with the face annotation boxes.

[0085] Step 15: Use a facial feature extraction model to extract facial feature data of each facial annotation box in the set of actions to be identified.

[0086] Step 16: Match the facial feature data of the target student with the pre-processed facial feature data in the feature database to obtain the identity information of the target student and the action information of the target student.

[0087] A student's facial feature data includes multiple facial feature vectors (such as frontal face feature vectors and side face feature vectors). Here, the facial feature data of the target student is matched with the preprocessed facial feature data in the feature database. First, the vectors are matched and then the identities are matched, which can improve the accuracy of identity recognition.

[0088] It can be seen that this application can match the spatiotemporal actions of students in the action tube set with the face annotation frames by performing spatial intersection operations on the action tube set and the face annotation frame set; match the facial feature data of the target student with the pre-processed facial feature data in the feature database to obtain the identity information of the target student and the action information of the target student, so that the spatiotemporal actions of the student are matched with the student's identity.

[0089] It should be noted that before executing the student spatiotemporal action recognition method provided in this application, the original video data needs to be preprocessed in order to obtain training data for various models in the student spatiotemporal action recognition method. The process is as follows:

[0090] Step a, data collection: mainly collect video data in classroom teaching scenes through cameras deployed in the classroom, extract frames and perform video segmentation processing on the video data to obtain image datasets and video datasets respectively.

[0091] Specifically, in order to ensure the authenticity of the collected video data and the spontaneity of students' actions during the collection process, data collection can be carried out in multiple classrooms to ensure the diversity of the environment in the video data, and classroom teaching of as many teaching subjects as possible can be collected to ensure the diversity of training samples.

[0092] The collected original video data are organized and classified in the form of classrooms and classroom teaching subjects. Each video is processed by extracting video RGB frames to obtain image data. The specific operation is to divide the video into video units according to the standard length of classroom teaching, extract RGB frames at intervals of 10 seconds in each video unit, and organize the obtained image data into original image datasets; each video is processed by video segmentation to obtain video data. The specific operation is to divide the video into 3-second video segments at every eighth of the video based on the actual length of the video, and organize the obtained video segments into original video datasets.

[0093] Step b, data definition and annotation: anonymize the student identities in the dataset, define the student classroom actions in combination with the teaching stage, and use annotation tools to annotate the face parts in the image dataset to obtain the student face dataset D f , the video dataset is processed into AVA format using a script (which conforms to the data organization format of the Atomic Visual Actions dataset, where the data labeling includes the character annotation box and its corresponding action category, and one frame is taken as a key frame per second in the video data for labeling), the students' bodies in the classroom are labeled and the corresponding spatiotemporal actions in the action table are defined, and the student body dataset D is obtained. b and student action dataset D a .

[0094] For example, before labeling the video data, the student identity information is anonymized, the identities of the students participating in the data collection process are numbered, and the student actions in the classroom teaching scene are defined and classified. According to the different teaching stages, a corresponding student action label vocabulary V is developed. a The default classroom teaching scene is primary education, and the student action label vocabulary includes listening carefully, discussing, raising hands, taking notes, etc. Select the clear picture data in the original picture dataset for annotation, and use the image annotation tool to annotate the original picture data. The annotation content includes student faces, facial key points and human bodies. During the annotation process, the student face annotation box is matched with the student identity number, and finally the organized student face dataset D is obtained. f With the student human data set D b .

[0095] Student face dataset D f Can be used as a training data set for face detection models, student human data set D b Can be used as a training data set for human detection models, student action data set D a It can be used as a training dataset for spatiotemporal action recognition models.

[0096] The original video dataset was processed in AVA format. First, the video was frame-extracted at 1 frame per second and 30 frames per second respectively. Then, the student body coordinates were generated by Faster-RCNN and imported into the annotation tool. The annotation box was manually corrected and the student action label vocabulary V was used. a Assign action labels to each annotation box, and finally obtain the student action video dataset D in AVA format a .

[0097] The specific process of step 11 (building a face feature extraction model based on the improved face recognition method and the image feature extraction neural network model) is exemplified below.

[0098] Step 11.1, calculate the formula

[0099]

[0100] The improved ArcFace loss function L is obtained; where N represents the size of each training batch, n represents the number of categories, m1, m2, and m3 represent the multiplicative angle margin penalty of SphereFace, the additive angle margin penalty of ArcFace, and the additive cosine margin penalty of CosFace, respectively. θ represents the angle between the feature vector of the training batch sample and the column vector of the weight matrix. The subscript j represents the number of columns in the weight matrix. i It represents the number of columns corresponding to the actual category of the feature vector of the i-th sample in the training batch, and s represents the radius of the hyperplane obtained by training.

[0101] In step 11.2, the pre-trained PyramidVision Transformer v2-B2-li neural network model is used as the image feature extraction neural network model, and the improved ArcFace loss function L is used to optimize the PyramidVisionTransformer v2-B2-li neural network model to obtain a face recognition feature extraction model.

[0102] The specific process of step 12 (building a spatiotemporal action recognition model based on the pre-trained human body detection model and spatiotemporal feature extraction neural network model) is exemplified below.

[0103] Step 12.1: Use the pre-trained human detection model to detect the original video data and obtain multiple human annotation boxes.

[0104] In step 12.2, ResNet50 is used as the backbone network of the SlowFast neural network model, and the SlowFast neural network model is used to extract the spatiotemporal features of the original video data and the motion information of each human body annotation box in multiple human body annotation boxes.

[0105] The above-mentioned SlowFast neural network model is a neural network model for video recognition.

[0106] Step 12.3: Build a spatiotemporal action recognition model based on spatiotemporal features and action information.

[0107] The following is an exemplary description of the specific process of performing a spatial intersection operation on the action tube set and the face annotation frame set in step 14 (performing a spatial intersection operation on the action tube set and the face annotation frame set to obtain a set of actions to be recognized).

[0108] In step 14.1, the action tube set A and the face annotation frame set F are respectively time-sequentially aligned with the video data to be processed to obtain the action tube set with timestamps and the face annotation frame set with timestamps.

[0109] Where A=A1,A2,...,A q ,F=F1,F2,...,F m , q represents the total number of action tubes, and m represents the total number of face annotation boxes.

[0110] Step 14.2: Double-crop the action tube set with timestamps to obtain a video frame set; the video frame set includes multiple action tubes with timestamps.

[0111] It is worth mentioning that trimming the action tube can simplify the processing object, reduce the amount of calculation and complexity, and improve the accuracy of recognition.

[0112] Step 14.2 specifically includes the following steps:

[0113] Step 14.2.1: Crop the action tube set containing the timestamp to obtain the first video slice.

[0114] Specifically, the first clipping is motion clipping, which clips the segments where the motion tube appears in the original video to obtain rough video slices.

[0115] Step 14.2.2: Divide the first video slice into regions using normal distribution to obtain multiple video slice regions.

[0116] Specifically, the second clipping is random normal clipping, which divides the rough video slices into regions using normal distribution.

[0117] Step 14.2.3: randomly extract video frames from multiple video slice areas to obtain a video frame set.

[0118] Specifically, video frames are randomly selected in the selected area, and the obtained video frames are organized and classified according to timestamps to obtain a video frame set.

[0119] Step 14.3: For each action annotation box in the video frame set, calculate the formula

[0120]

[0121] h inter =max(0,min(ay2,by2)-max(ay1,by1))

[0122] w inter =max(0,min(ax2,bx2)-max(ax1,bx1))

[0123] Get the intersection over union (IoU) of the action annotation box and all face annotation boxes at the same timestamp; where the coordinates of the action annotation box are (ax1, ay1, ax2, ay2), and the coordinates of the face annotation box are (bx1, by1, bx2, by2), IoU = IoU1, IoU2, ..., IoU g , g represents the total number of face annotation boxes corresponding to the action annotation box at the same timestamp.

[0124] Step 14.4: The face annotation frame with the highest intersection-with-union ratio and the closest geometric distance to the center point of the action annotation frame is taken as the initiator of the action in the action annotation frame, thereby obtaining a set of face action data.

[0125] Step 14.5: construct a set of actions to be recognized based on multiple groups of facial action data corresponding to all action annotation boxes.

[0126] The following is an exemplary explanation of the specific process of matching the target student's facial feature data with the preprocessed facial feature data in the feature database in step 16 (matching the target student's facial feature data with the preprocessed facial feature data in the feature database to obtain the target student's identity information and the target student's action information).

[0127] Step 161, by calculating the formula

[0128]

[0129] The cosine similarity cosθ between the target student’s facial features and the pre-processed facial features in the feature database is obtained, where the target student’s facial features are ax i Represents the i-th feature component of the target student’s facial features. The pre-processed facial features in the feature database are bx i It represents the i-th feature component of the preprocessed facial features in the feature database, and n represents the total number of feature components of the student's facial features.

[0130] Step 16.2: If cosθ is greater than the preset cosine similarity threshold, it is determined that the target student's facial features match the pre-processed facial features, and the identity information corresponding to the pre-processed facial features in the feature database is determined as the identity information of the target student.

[0131] Exemplarily, the preset threshold of cosine similarity is set to α. When cosθ>α, the facial features are determined to match, and then identity matching is achieved. It should be noted here that multiple pre-processed facial features can point to the same student identity; otherwise, the facial features are determined to be mismatched, and the cosine similarity with other pre-processed facial features in the feature database is continued to be calculated.

[0132] Step 16.3: Determine the action information in the action annotation box corresponding to the facial feature data of the target student in the action set to be identified as the action information of the target student.

[0133] The following is an illustrative description of the process of processing a video using the student spatiotemporal action recognition method provided by this application in conjunction with specific embodiments.

[0134] like Figure 3 As shown, in one embodiment of the present application, the pre-trained face detection model and spatiotemporal action recognition model are first used to process the video data to be processed to obtain an action tube set and a face annotation frame set. The action tube set A and the face annotation frame set F are then time-aligned with the video data to be processed to obtain an action tube set with timestamps and a face annotation frame set with timestamps. The action tube set with timestamps is then double-cropped to obtain a video frame set. The action tube set and the face annotation frame set are then spatially intersected to obtain an action set to be identified. The facial feature extraction model is then used to extract the facial feature data of each face annotation frame in the action set to be identified. Finally, the facial feature data of the target student is matched with the pre-processed facial feature data in the feature database to obtain the identity information of the target student and the action information of the target student.

[0135] The student spatiotemporal action recognition method provided by this application has the following advantages:

[0136] 1. By adopting cutting-edge technical models to complete technology iteration and performance improvement, inspired by transfer learning, the model is pre-trained on open source and difficult datasets, and then trained on actual datasets to obtain a usable model with good generalization ability, fast detection speed and high detection accuracy.

[0137] 2. Using a face verification method based on an improved ArcFace, the intra-class compactness and inter-class diversity of the learned facial representations are significantly improved. Combined with PVT-v2-b2-li, a large-scale identity verification model that can support the recognition of millions of identities / entities can be obtained.

[0138] 3. Different from traditional frame-level motion detection, the student spatiotemporal motion recognition method provided in this application combines the student body detection model and the spatiotemporal motion recognition model to achieve reliable spatiotemporal motion detection of students in classroom teaching scenarios.

[0139] 4. Different from the previous method of collecting and using identity and motion data separately, the student spatiotemporal motion recognition method provided by this application can simultaneously identify and match the identity and motion of students, realize the collection of information on individual students, and the obtained information is stored in the database, which can provide data support for subsequent applications such as student personalized analysis and teaching plan formulation.

[0140] The student spatiotemporal motion recognition device provided by this application is exemplarily described below in conjunction with specific embodiments.

[0141] like Figure 2 As shown, an embodiment of the present application provides a student spatiotemporal motion recognition device, the student spatiotemporal motion recognition device 200 includes:

[0142] The face feature module 201 is used to construct a face feature extraction model based on the improved face recognition method and the image feature extraction neural network model.

[0143] The spatiotemporal action recognition module 202 is used to construct a spatiotemporal action recognition model based on a pre-trained human body detection model and a spatiotemporal feature extraction neural network model;

[0144] The video processing module 203 is used to process the video data to be processed using the pre-trained face detection model and spatiotemporal action recognition model to obtain an action tube set and a face annotation frame set; the action tube set includes multiple action annotation frame sets.

[0145] The spatial intersection module 204 is used to perform a spatial intersection operation on the action tube set and the face annotation frame set to obtain a set of actions to be recognized; the set of actions to be recognized includes multiple face annotation frames and an action annotation frame corresponding to each face annotation frame.

[0146] The facial feature extraction module 205 is used to extract facial feature data of each facial annotation frame in the set of actions to be identified using a facial feature extraction model; the facial feature data includes multiple facial features.

[0147] The facial feature matching module 206 is used to match the facial feature data of the target student with the pre-processed facial feature data in the feature database to obtain the identity information of the target student and the action information of the target student.

[0148] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0149] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0150] like Figure 4 As shown, an embodiment of the present application provides a terminal device, and the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 4 Only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 implements the steps of any of the above method embodiments when executing the computer program D102.

[0151] Specifically, when the processor D100 executes the computer program D102, it constructs a face feature extraction model based on the improved face recognition method and the image feature extraction neural network model; constructs a spatiotemporal action recognition model based on the human body detection model and the spatiotemporal feature extraction neural network model; uses the face detection model and the spatiotemporal action recognition model to process the video data to obtain an action tube set and a face annotation frame set; performs spatial intersection on the action tube set and the face annotation frame set to obtain a set of actions to be identified; uses the face feature extraction model to extract the face feature data of each face annotation frame; matches the face feature data of the target student with the preprocessed face feature data to obtain the identity information and action information of the target student, thereby matching the student's spatiotemporal actions with the student's identity.

[0152] The processor D100 may be a central processing unit (CPU), or may be another general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0153] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk, a smart memory card (SMC, SmartMedia Card), a secure digital (SD, Secure Digital) card, a flash card, etc. equipped on the terminal device D10. Furthermore, the memory D101 may also include both an internal storage unit of the terminal device D10 and an external storage device. The memory D101 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory D101 may also be used to temporarily store data that has been output or is to be output.

[0154] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0155] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0156] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the student's spatiotemporal motion recognition device / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, mobile hard disk, magnetic disk or optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0157] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0158] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0159] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0160] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0161] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for identifying students' spatiotemporal actions, characterized in that: include: Construct a facial feature extraction model based on the improved face recognition method and image feature extraction neural network model; Construct a spatiotemporal action recognition model based on the pre-trained human detection model and spatiotemporal feature extraction neural network model; Processing the video data to be processed using the pre-trained face detection model and the spatiotemporal action recognition model to obtain an action tube set and a face annotation frame set; the action tube set includes multiple action annotation frame sets; Performing a spatial intersection operation on the action tube set and the face annotation frame set to obtain a set of actions to be recognized; the set of actions to be recognized includes multiple face annotation frames and an action annotation frame corresponding to each face annotation frame; The performing a spatial intersection operation on the action tube set and the face annotation frame set includes: The action management sets are respectively and the face annotation box set , and the video data to be processed are time-series aligned to obtain a set of action tubes with timestamps and a set of face annotation frames with timestamps; wherein, , , Indicates the total number of action pipes, Indicates the total number of face annotation boxes; Double cropping the action tube set with the timestamp to obtain a video frame set; the video frame set includes multiple action tubes with the timestamp; For each action annotation frame in the video frame set, the formula ； ； ； ； Get the intersection of the action annotation box and all face annotation boxes at the same timestamp ; Among them, the coordinates of the action annotation box are , the face annotation frame coordinates are , , Indicates the total number of face annotation boxes corresponding to the action annotation box at the same timestamp; The face annotation frame with the highest intersection-with-union ratio and the closest geometric distance to the center point of the action annotation frame is taken as the initiator of the action in the action annotation frame, and a set of face action data is obtained; Constructing the set of actions to be recognized based on multiple groups of facial action data corresponding to all action annotation boxes; Extracting facial feature data of each face annotation frame in the set of actions to be identified using the facial feature extraction model; the facial feature data includes a plurality of facial features; The facial feature data of the target student is matched with the pre-processed facial feature data in the feature database to obtain the identity information of the target student and the action information of the target student.

2. The identification method according to claim 1, characterized in that The face feature extraction model is constructed based on the improved face recognition method and the image feature extraction neural network model, including: By calculating the formula ； Get the improved ArcFace loss function ;in, represents the size of each training batch, represents the number of categories, 、 、 They represent the multiplicative angle margin penalty of SphereFace, the additive angle margin penalty of ArcFace, and the additive cosine margin penalty of CosFace, respectively. Represents the angle between the training batch sample feature vector and the weight matrix column vector, subscript represents the number of columns of the weight matrix, Represents the training batch The number of columns corresponding to the actual category of the sample feature vector, represents the radius of the hyperplane obtained by training; The pre-trained Pyramid Vision Transformer v2-B2-li neural network model is used as the image feature extraction neural network model, and the improved ArcFace loss function is used. The Pyramid Vision Transformer v2-B2-li neural network model is optimized and trained to obtain the face recognition feature extraction model.

3. The identification method according to claim 1, characterized in that The spatiotemporal action recognition model is constructed based on the pre-trained human body detection model and the spatiotemporal feature extraction neural network model, including: Detecting the original video data using the pre-trained human body detection model to obtain multiple human body annotation frames; ResNet50 is used as the backbone network of the SlowFast neural network model, and the SlowFast neural network model is used to extract the spatiotemporal features of the original video data and the motion information of each human body annotation frame in the multiple human body annotation frames; The spatiotemporal action recognition model is constructed according to the spatiotemporal features and the action information.

4. The identification method according to claim 1, wherein: The double cropping of the action tube set containing the timestamp to obtain a video frame set includes: Cutting the action tube set containing the timestamp to obtain a first video slice; Dividing the first video slice into regions using normal distribution to obtain a plurality of video slice regions; Video frames are randomly extracted from the multiple video slice areas to obtain the video frame set.

5. The identification method according to claim 1, characterized in that The matching of the target student's facial feature data with the pre-processed facial feature data in the feature database includes: By calculating the formula ； Get the cosine similarity between the target student's facial features and the pre-processed facial features in the feature database , where the target student’s facial features are , The first feature components, the pre-processed face features in the feature database are , Represents the first pre-processed facial feature in the feature database feature components, The total number of feature components representing the student's facial features; like If the cosine similarity is greater than a preset threshold, it is determined that the target student's facial features match the pre-processed facial features, and the identity information corresponding to the pre-processed facial features in the feature database is determined as the identity information of the target student; The action information in the action annotation box corresponding to the facial feature data of the target student in the set of actions to be identified is determined as the action information of the target student.

6. The identification method according to claim 1, characterized in that After constructing the face feature extraction model according to the improved face recognition method and the image feature extraction neural network model, the recognition method further includes: The facial feature extraction model is used to extract facial feature data of all students, and a feature database is constructed based on the facial feature data of all students.

7. A student spatiotemporal motion recognition device, characterized in that: include: A facial feature module is used to construct a facial feature extraction model based on the improved face recognition method and the image feature extraction neural network model; The spatiotemporal action recognition module is used to build a spatiotemporal action recognition model based on the pre-trained human body detection model and spatiotemporal feature extraction neural network model; A video processing module is used to process the video data to be processed using a pre-trained face detection model and the spatiotemporal action recognition model to obtain an action tube set and a face annotation frame set; the action tube set includes multiple action annotation frame sets; a spatial intersection module, configured to perform a spatial intersection operation on the action set and the face annotation frame set to obtain a set of actions to be recognized; the set of actions to be recognized includes a plurality of face annotation frames and an action annotation frame corresponding to each face annotation frame; The specific process of the spatial intersection module includes: The action management sets are respectively and the face annotation box set , and the video data to be processed are time-series aligned to obtain a set of action tubes with timestamps and a set of face annotation frames with timestamps; wherein, , , Indicates the total number of action pipes, Indicates the total number of face annotation boxes; Double cropping the action tube set with the timestamp to obtain a video frame set; the video frame set includes multiple action tubes with the timestamp; For each action annotation frame in the video frame set, the formula ； ； ； ； Get the intersection of the action annotation box and all face annotation boxes at the same timestamp ; Among them, the coordinates of the action annotation box are , the face annotation box coordinates are , , Indicates the total number of face annotation boxes corresponding to the action annotation box at the same timestamp; The face annotation frame with the highest intersection-with-union ratio and the closest geometric distance to the center point of the action annotation frame is taken as the initiator of the action in the action annotation frame, and a set of face action data is obtained; Constructing the set of actions to be recognized based on multiple groups of facial action data corresponding to all action annotation boxes; A facial feature extraction module, configured to extract facial feature data of each face annotation frame in the set of actions to be identified using the facial feature extraction model; the facial feature data includes a plurality of facial features; The facial feature matching module is used to match the facial feature data of the target student with the pre-processed facial feature data in the feature database to obtain the identity information of the target student and the action information of the target student.

8. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the student spatiotemporal action recognition method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for identifying spatiotemporal actions of students as described in any one of claims 1 to 6 is implemented.

Images