Gesture feature and regression task assisted fine-grained student classroom behavior identification method

Abstract

The invention relates to a fine-grained student classroom behavior recognition method assisted by attitude features and regression tasks, and belongs to the field of computer vision. According to the method, 26 types of student fine-grained classroom behaviors are firstly defined, five characteristics of joint angles, postures and the like are extracted according to the characteristics that the classroom behaviors of the students are fine and smooth in action and the behaviors of the different students have relevance, the relevance among the behaviors of the different students is automatically analyzed by adopting a Transformers framework, and a posture regression task is used as assistance, so that the classroom behaviors of the students can be automatically analyzed. And a student classroom behavior recognition deep learning model is constructed. By inputting the multi-dimensional features of the student video, one or more behaviors of each student in the video can be identified. The student fine-grained classroom behavior recognition method provided by the invention has important significance in the field of intelligent education research, and the constructed behavior recognition model obtains similar or higher recognition precision with a baseline model on a data set of the invention and a public behavior recognition data set.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a posture feature and regression task-assisted fine-grained student classroom behavior recognition method. Background Art

[0002] Identification of student classroom behavior is an important basis for studying students' learning interests, personality traits, and knowledge mastery, and is also the main research content of "student-centered" smart education. Due to the restrictions of class discipline and public environment, students will have the pressure of seeking knowledge, the joy of harvest, and the need for communication during the learning process. Therefore, students' classroom behaviors are rich and subtle. The more delicate movements of students' heads, eyes, necks, torsos, etc. during class reflect a rich psychological state. For example, when students are in a state of high concentration, their heads usually lean forward slightly and their eyes look straight ahead. When they are nervous or anxious, students may move their eyes frequently to avoid eye contact with the teacher, while their heads slightly droop. When students feel bored or absent-minded, their heads may turn frequently, their eyes wander around the classroom or outside the window, and it is difficult to focus on the teacher or the textbook. In addition, the angle changes of students' skeletal joints are closely related to the categories of classroom behavior, and different classroom behaviors are usually accompanied by specific body postures and movements. For example, when students write or take notes, their bodies usually lean forward, their arms are close to the table, and they maintain a certain angle to stabilize the handwriting. At this time, the forward tilt angle of the spine and the bending angles of the elbows and wrists are important features for identifying such behaviors. Therefore, in order to have a deeper understanding of the psychological state of students during class, it is necessary to define and identify the delicate movements of students' heads, necks, torsos, eyes, hands, etc. However, the current recognition of student classroom behaviors mainly focuses on individual or general movements such as "raising the head", "raising hands", and "sleeping", and lacks the definition of delicate movements and related data sets.

[0003] Human behavior recognition is a major research topic in computer vision. At present, human behavior recognition methods based on deep learning have made great progress compared to methods based on traditional machine learning and have become the mainstream method for human behavior recognition. Compared with other scenarios, human behavior in classroom scenarios has its own unique characteristics. In a classroom scenario, multiple students are listening to the same class, and their behaviors are naturally connected. At the same time, due to different individual psychological states, the behaviors of different students are more or less different. Therefore, compared with human behavior recognition in other scenarios, the student classroom behavior recognition task has different focuses and difficulties. This task needs to consider the connections and differences between different individuals, and also needs to distinguish different behaviors with very small differences. Summary of the invention

[0004] The purpose of the present invention is to provide a fine-grained student classroom behavior recognition method assisted by posture features and regression tasks. The method defines 26 types of student classroom behaviors and realizes the fine-grained classification of student classroom behaviors, which is an important basis for analyzing students' psychological states. In addition, the method fully considers the important role of features such as bone joints and postures in the fine-grained classification of behaviors, and uses the Transformers architecture to analyze the correlation between the behaviors of different students. At the same time, the posture regression task is used as an auxiliary to effectively improve the accuracy of fine-grained student classroom behavior recognition. Experimental results show that the multi-task assistance, multi-feature fusion, and extraction of correlation information of different individual behaviors proposed by the method are beneficial to solving the technical difficulties that are easy to confuse in fine-grained behavior recognition.

[0005] The above-mentioned object of the present invention is achieved by the following technical solutions:

[0006] The fine-grained student classroom behavior recognition method assisted by posture features and regression tasks first defines 26 common and refined student classroom behaviors and constructs a student classroom behavior dataset. On this basis, the 3D joint angle features, posture features (2D joint angle features, global body direction features, position offset features) and video appearance features of the student body parts in the video are extracted through the corresponding algorithm, and a fine-grained student classroom behavior recognition model assisted by posture features and regression tasks is designed. By inputting the multi-dimensional features of the student video, one or more behaviors of each student in the video can be recognized. The following steps are included:

[0007] Step 1: Define student classroom behaviors: When selecting student classroom behavior categories, 26 types of student classroom behaviors were defined based on student body changes, including head, body, sight, hands, and mouth, including 7 head behaviors, 4 body behaviors, 5 sight behaviors, 9 hand behaviors, and 1 mouth behavior;

[0008] Step 1.1: Head behavior definition, select 7 head-based student classroom behaviors and define them according to facial orientation, head tilt, chin and neck angle, and gaze direction;

[0009] Step 1.2: Definition of body behaviors. Four student classroom behaviors based on the upper body were selected and defined according to body angle, head orientation, and shoulder height.

[0010] Step 1.3: Definition of gaze behavior: select 5 types of student classroom behaviors based on gaze and define them according to the gaze focus;

[0011] Step 1.4: Hand behavior definition, select 9 student classroom behaviors based on hand states and hand-object interactions, and define them according to hand states;

[0012] Step 1.5: Mouth behavior definition. Select 1 mouth-based student classroom behavior and define it according to the mouth state.

[0013] Step 2: Establish a website and method for annotating students’ classroom behaviors, and construct a dataset of students’ classroom video behaviors; design the criteria for dataset annotation and the website for annotating video data based on the definition of students’ classroom behaviors and the collected students’ classroom videos;

[0014] Step 2.1: Dataset Annotation Criteria,To ensure the high quality of data annotation, the following three measures,are taken: annotator selection, full supervision, and double annotation;

[0015] Step 2.2: Design of a data annotation website. Design an online data annotation website. During the data annotation process, the administrator uploads the video samples to be annotated to the online website. The annotators can watch the videos directly on the website and annotate the behavior categories of the students in the videos by single or multiple selection. The data annotation website includes the following five parts: data table management, basic information management, automatic annotation of the student positions in the video, fine-tuning of the student positions in the video, and annotation of the student behaviors in the video.

[0016] Step 2.3: Dataset Statistics,During the creation of the new dataset, a total of 66 videos of students attending lectures,,from these videos, 1906 2-second student video clips with frequent,changes in student behaviors were selected, which contained a total of 54,461,student behavior annotations.

[0017] Step 3: Extraction and fusion of multiple features: Extract five features related to students' classroom behavior, namely, 3D joint angle features, 2D joint angle features of posture features, global body direction features, position offset features, and video appearance information, and design a multi-feature fusion method based on deep learning; the extraction and fusion method of the multiple features is as follows:

[0018] Step 3.1: 3D joint angle feature extraction, using the PHALP (Predicting Human Appearance, Location and Pose for Tracking, referred to as PHALP) algorithm to extract the 3D joint angle information of the person in each frame of the video i , where J i ∈R T×135 , T represents the length of the video, i represents the i-th student appearing in the video;

[0019] Step 3.2: Posture feature extraction: Use the SMPL (Skinned Multi-Person Linear, SMPL for short) model to extract the student's posture feature representation in each frame of the video; this includes the 2D joint angle feature θ i ∈R T×23×3×3 (23 human body joints are represented by 3×3 vectors respectively), global direction features of the body And the position offset feature ω i ∈R T×10 , where T represents the length of the video, and i represents the i-th student appearing in the video;

[0020] Step 3.3: Video appearance feature extraction: Use the PHALP algorithm to track and obtain the trajectory of each student in the video, run the pre-trained 2D action recognition model MViT (Multiscale Vision Transformers, MViT for short), and save its feature vector before the classification layer as the video appearance feature of each student in the video, i.e., A i ∈R T×4096 ;

[0021] Step 3.4: Multi-feature fusion. Through steps 3.1 to 3.3, multiple feature representations of students in the video are obtained; the features extracted using SMPL in step 3.2 are linearly spliced ​​in the time dimension as the posture feature representation of the i-th student in the video. The posture features of N students in the video are expressed as P = [P1,…,P i ,…P N ]∈R (N×T)×226 ; Furthermore, the feature of the i-th student in a video is represented by G i =[J i ,P i ,A i ];

[0022] First, two linear networks are used to 3D joint angle information J i Encoding, the formula is:

[0023]

[0024] Among them, J i ∈R T×135 Represents the 3D joint angle information of the i-th student in the video, using the ReLU activation function; Encode network parameters for 3D joint angle information; The encoding features representing the 3D joint angle information of the i-th student;

[0025] Next, a linear network is used to train the pose feature Pi Encoding, the formula is:

[0026]

[0027] Among them, P i ∈R T×226 Represents the posture information of the i-th student in the video; Encode network parameters for pose information; The encoding features representing the posture information of the i-th student;

[0028] Secondly, use the linear network and the bidirectional GRU network to analyze the appearance feature A i Encode, for the appearance feature A i ∈R T ×4096 Each time step feature x t ∈R 1×4096 Perform forward GRU calculation, the formula is:

[0029]

[0030] For appearance feature A i ∈R T×4096 Each time step feature x t ∈R 1×4096 Perform reverse GRU calculation, the formula is:

[0031]

[0032] Among them, σ(·) represents the Sigmoid activation function, ⊙ represents the matrix product, is the input weight matrix, is the weight matrix of the hidden state at the previous time step, are the parameters of GRU; Represents the hidden state of the feature at the tth moment of the reverse GRU network;

[0033] At each time step t, the final hidden state h of the bidirectional GRU t is the forward hidden state and the reverse hidden state connection, that is Appearance Features A i The final output of the bidirectional gated recurrent neural network is h = [h1,h2…,h T ]∈R T×8192 ;

[0034] Next, a linear network is used to encode the GRU output feature h of the appearance feature, and the formula is:

[0035]

[0036] in, Encode network parameters for appearance features; The encoding features representing the posture information of the i-th student;

[0037] Finally, the encoding features of the 3D joint angles Posture encoding features And appearance characteristics Perform linear splicing in the time dimension to obtain fusion features The formula is;

[0038]

[0039] in, represents feature concatenation, represents the fusion coding features of the i-th student; T represents the time length of the video;

[0040] Step 3.5: Fusion of multi-person implicit relationship features: Use a multi-layer Transformer network structure to fuse the relationship features of potential associations; multi-person implicit relationship feature fusion is mainly divided into the following two modules: location information extraction and Transformer fusion.

[0041] Location information extraction: Use the YOLOv7 model to detect the student position in each frame of the video and obtain the location information of N students in the video. Locations = {L1, L2, ... L i ,…,L N}, where L i =[left i ,top i ,right i ,bottom i ]∈R 1×4 ,left i Indicates the relative pixel distance between the detected left border of the student box and the left border of the video. i Indicates the relative pixel distance between the detected upper border of the student box and the upper border of the video, right i Indicates the relative pixel distance between the right border of the detected student box and the right border of the video, bottom i Indicates the relative pixel distance between the detected lower border of the student box and the lower border of the video;

[0042] Next, the position encoding feature L i and student characteristics Perform the splicing operation, the formula is:

[0043]

[0044] in represents the feature representation of the i-th student in the video with position embedding; Represents the concatenation of feature vectors in the time dimension.

[0045] Transformer fusion, given a student classroom video V, whose scene contains information about N students, through the feature encoding step, the video feature representation is obtained Among them, F∈R N×T×1028 , transform the video feature representation F to satisfy the input of the Transformer network, that is, F∈R 1 ×(N×T)×1028 , the formula is:

[0046]

[0047] Among them, Transformers(·) represents a multi-layer Transformer structure, the number of layers is set to 12, and the number of hidden layer units is set to 512; Represents the result of N student features in a given video being encoded by the Transformers network.

[0048] Step 4: Build a multi-task decoder: During the decoding process, design a posture estimation decoder and a video behavior recognition decoder for posture estimation and video behavior recognition tasks respectively. The specific steps are as follows:

[0049] Step 4.1 Pose Estimation Decoder. The pose estimation decoder contains three submodules, which respectively extract the three pose features extracted by the SMPL algorithm, the 2D joint angle feature θ i , global orientation characteristics of the body And the position offset feature ω i Conduct regression training;

[0050] For the 2D joint angle features θ of all students in the video, a multi-layer perceptron network is used for decoding; the formula is:

[0051]

[0052] Where σ(·) represents the Sigmoid activation function; It represents the feature representation of the N initial features of students in the video after being encoded by the encoding layer; Decode network parameters for 2D joint angles; Decode network parameters for 2D joint angles; Represents the 2D joint angle decoded feature representation of N students in the video;

[0053] Global body orientation features for all students in the video A multi-layer perceptron network is used for decoding; its formula is:

[0054]

[0055] in, represents the feature representation of the N initial features of students in the video after being encoded by the encoding layer; σ(·) represents the Sigmoid activation function; Decoding network parameters for global body orientation; Decoded representation of the global orientation features of the bodies of N students in the video;

[0056] For the position offset features ω of all students in the video, a multi-layer perceptron network is used for decoding; the formula is:

[0057]

[0058] in, represents the feature representation of the N initial features of students in the video after being encoded by the encoding layer; σ(·) represents the Sigmoid activation function; Decode network parameters for position offset; Decoded representation of the position offset features of N students in the video;

[0059] Finally, the three decoded features are concatenated in the second dimension to obtain the final posture feature decoding representation

[0060] Step 4.2 Video behavior classification decoder; Video behavior classification decoder is used for decoding behavior recognition; 26 binary classifiers are used to identify whether the target student contains the corresponding 26 behaviors; the output of the model encoding layer for the nth time in the video and the student at the tth time Perform behavior recognition decoding. The specific decoding formula for the student behavior that includes behavior category c is:

[0061]

[0062] in, represents the decoded representation of the nth student in the video V at the tth time on the category c; represents the fused feature representation of the nth student in video V at time t, 512 represents the dimension of the feature; σ(·) represents the Sigmoid activation function; is the decoding network parameter of category c, where c represents one of the 26 student classroom behavior categories; finally, the decoding of the behavior recognition decoder of the nth student at the tth moment in the video on the 26 behavior recognitions is expressed as

[0063]

[0064] The decoding results of the nth student in the video at the tth time on the 26 behaviors are normalized, and the formula is:

[0065]

[0066] in, It is the feature representation obtained by normalizing the decoded features on 26 behavior categories, n represents the nth student in the video, and t represents the tth moment in the video; represents the probability that the student’s decoding behavior contains behavior category c1, represents the probability that the student's decoding behavior does not include the behavior category c1; the probability of the nth student's decoder in the 26 behavior categories at the tth moment in the final video V is,

[0067] Step 4.3 Multi-task loss function setting: the overall target loss of the multi-task model is obtained by weighted addition of the losses of the two tasks of behavior recognition and posture estimation. Among them, the posture estimation regression task uses the mean square error loss function to calculate its loss, and the behavior recognition task uses the cross entropy loss function to calculate its loss.

[0068] The pose estimation regression task uses the mean square error loss function to calculate the model loss; specifically, the true pose feature of the nth student in the video at time t is represented by P n,t ∈R 1×226 , the feature representation of the model decoding of the nth student in the video at time t is n represents the nth student in the video, t represents the tth moment in the video; the loss calculation formula for the posture estimation regression task is:

[0069]

[0070] Among them, L pose is the loss value of the pose estimation task of the nth student in the video at the tth time, P n,t ∈K 1×226 represents the posture feature label of the nth student in the video at the tth moment; It represents the decoded features of the posture decoder of the nth student in the video at the tth moment;

[0071] The student behavior recognition task uses the cross entropy loss function to calculate the loss, as follows: The feature representation of the nth student in the video at the tth time is decoded by the model: The true label y of the nth student in the video at time t n,t ∈R 1×26 ,y n,t (i) is the vector y n,t The i-th element of :

[0072]

[0073] The loss calculation formula for a single student at time t is:

[0074]

[0075] Finally, the overall loss of the model is calculated as:

[0076] L=αL pose +(1-α)L action (twenty four)

[0077] L is the overall loss function value of the multi-task, and α is a hyperparameter used to balance the importance of the two tasks and is set to 0.3.

[0078] Step 5: Design evaluation indicators and experimental parameters to implement multi-task model training;

[0079] Step 6: Design a comparative experiment to test the usefulness of the method proposed in the present invention.

[0080] The beneficial effects of the present invention are as follows: the student classroom behavior recognition method provided by the present invention takes into account the important auxiliary role of body posture features and posture regression tasks in student classroom behavior recognition, integrates five features: 3D joint angle features, 2D joint angle features, body global direction features, position offset features, and video appearance features, and adopts a multi-task model to realize behavior recognition. In addition, in individual behavior recognition, the connection between the behavior of the individual and the behavior of different individuals around is also considered. Student classroom behavior recognition is an important clue to understanding the cognitive participation and emotional participation of students in the learning process, and is the basis for establishing "student-centered" smart education. Based on student classroom behavior recognition, information on students' interests, psychological states, and the degree of mastery of relevant knowledge points can be obtained, and then "classroom portraits" of individual students and "classroom portraits" of group students can be constructed.

[0081] The present invention defines 26 types of student classroom behaviors, constructs a video dataset containing 54,461 student classroom behaviors, and builds a model to realize the recognition of corresponding behaviors, which is of great significance to the field of smart education research. Moreover, the constructed behavior recognition model has achieved recognition accuracy similar to or higher than that of the baseline model on the dataset of the present invention and the public behavior recognition dataset. Compared with the prior art, the present invention has the following advantages: First, it fully considers the important auxiliary role of body posture characteristics and posture regression tasks, and effectively solves the technical difficulty that fine-grained behaviors are easily confused in recognition. Second, it considers the correlation between different individual behaviors and utilizes the clues provided by the environment, which is beneficial to supplement the necessary information for individual behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] The drawings described herein are used to provide further understanding of the present invention and constitute a part of this application. The illustrative examples of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0083] Figure 1 is a flow chart of the method of the present invention;

[0084] Figure 2 This is a diagram of the coding layer structure of the model proposed in the present invention;

[0085] Figure 3 This is a structural diagram of the decoding layer of the model proposed in the present invention;

[0086] Figure 4 This is a sample diagram of the system test of the proposed model of the present invention. DETAILED DESCRIPTION

[0087] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0088] See also Figures 1 to 4As shown in the figure, the fine-grained student classroom behavior recognition method assisted by posture features and regression tasks of the present invention first proposes a student classroom behavior definition standard based on student limb changes (head, torso, line of sight, hands and mouth), covering 26 different student classroom behaviors. Secondly, a fine-grained student classroom behavior recognition model assisted by posture features and regression tasks is designed. The model extracts 3D joint angle features, posture features (2D joint angle features, global body direction features, position offset features), and video appearance features of student body parts in the video through corresponding algorithms, and then effectively fuses multiple student features to obtain rich student feature representations. Finally, a multi-task training framework based on posture estimation regression and video behavior classification is designed to realize the recognition of one or more behaviors of each student in the classroom in the video.

[0089] The posture feature and regression task-assisted fine-grained student classroom behavior recognition method of the present invention constructs a student classroom behavior recognition dataset, effectively integrates five features related to student classroom behavior, and improves the recognition performance of the model.

[0090] Step 1: Define student classroom behaviors. When selecting student classroom behavior categories, the present invention defines 26 common student classroom behaviors based on student body changes (head, body, sight, hand, mouth), including 7 head behaviors, 4 body behaviors, 5 sight behaviors, 9 hand behaviors, and 1 mouth behavior.

[0091] Step 1.1: Head behavior definition. The present invention selects 7 head-based student classroom behaviors and defines them according to facial orientation, head inclination, chin and neck angle, and line of sight direction.

[0092] Raise your head: Keep your head level, face facing forward, the angle between your head and shoulders close to 90 degrees, and your eyes looking straight ahead.

[0093] Lower your head: face downward, the angle between chin and neck is less than 90 degrees, head tilted forward, neck bent forward, eyes looking downward.

[0094] Head tilted back: face facing upwards, the angle between chin and neck is greater than 90 degrees, head tilted back, neck tilted back, eyes looking upwards.

[0095] Left head tilt: The face faces forward and the head tilts to the left so that the angle between the head and the left shoulder is less than 90 degrees. The neck is bent to the left and the eyes remain looking straight ahead or slightly adjusted with the tilt of the head.

[0096] Right head tilt: The face faces forward and the head tilts to the right so that the angle between the head and the right shoulder is less than 90 degrees. The neck is bent to the right and the eyes remain looking straight ahead or slightly adjusted with the tilt of the head.

[0097] Turn head left: face facing left, the angle between head and left shoulder is close to 90 degrees, head is perpendicular to the paper and facing inward, neck twists with the rotation of the head, and eyes turn to the left with the face.

[0098] Turn head right: face facing right, the angle between head and right shoulder is close to 90 degrees, the head is perpendicular to the paper and facing inward, the neck twists with the rotation of the head, and the eyes turn to the right with the face.

[0099] Step 1.2: Body behavior definition. The present invention selects 4 types of student classroom behaviors based on the upper body, and defines them according to body angle, head inclination, and shoulder height.

[0100] The body is tilted to the left: the upper torso is straight but overall deviated to the left, the left shoulder is lower than the right shoulder, and the shoulder line is not parallel to the ground.

[0101] The body is tilted to the right: the upper torso is straight but overall deviated to the right, the left shoulder is higher than the right shoulder, and the shoulder line is not parallel to the ground.

[0102] Body Straight: The entire upper torso and head remain straight, with shoulders stable and without leaning or movement.

[0103] Lean forward: The entire upper torso tilts forward, with the center of gravity located in front, and the body appears above the table.

[0104] Step 1.3: Definition of sight line behavior. The present invention selects 5 types of student classroom behaviors based on sight line and defines them according to the sight line focus.

[0105] Eyes looking forward: The eyes are focused on the target or direction directly ahead.

[0106] Eyes directed toward another person: The gaze or focus of the eyes is on another person.

[0107] Eyes on the phone: The focus or gaze of the eyes is on the phone screen.

[0108] Eyes look at the table: The focus or line of sight of the eyes is on the table.

[0109] Eyes look under the table: the focus or line of sight of the eyes is concentrated under the table top or on the ground.

[0110] Step 1.4: Hand behavior definition. The present invention selects 9 student classroom behaviors based on hand states and hand-object interactions, and defines them according to the hand states.

[0111] Hand holding a pen: the state in which fingers hold or touch the pen.

[0112] Hands on head: the state of supporting or holding the head or part of the face with hands.

[0113] Writing: The process of writing using a pen or other writing instrument.

[0114] Crossing arms: The state of crossing arms in front of the chest or body.

[0115] Hand touching head: the process of touching the head or hair with hands.

[0116] Hand to face: The process of touching the face with hands.

[0117] Holding a mobile phone: A posture in which the hand holds or supports the mobile phone.

[0118] Holding glasses with hands: the state of touching, adjusting or supporting glasses with hands.

[0119] Hand tap click: An action in which the hand moves quickly and touches the target surface.

[0120] Step 1.5: Mouth behavior definition: The present invention selects one type of student classroom behavior based on the mouth and defines it according to the mouth state.

[0121] Talking with others: The process of exchanging information, interacting and communicating with another person through verbal and non-verbal signals, including both speaking and listening.

[0122] Step 2: Establish a website and method for annotating students’ classroom behaviors, and construct a dataset of students’ classroom video behaviors. Based on the definition of students’ classroom behaviors and the collected students’ classroom videos, a dataset annotation criterion and a video data annotation website were designed.

[0123] Step 2.1: Dataset Annotation Guidelines. To ensure high quality of data annotation, the following three measures were taken:

[0124] (1) Selection of labelers: A set of clear labeling rules is set for each volunteer, and a trial labeling of a few video samples is performed first. Only volunteers who meet the labeling standards can participate in the labeling of the entire dataset.

[0125] (2) Full supervision: After selecting qualified labelers, assign sample videos to each labeler and provide full supervision and guidance throughout the labeling process.

[0126] (3) Double labeling: Each sample video is labeled by two different labelers.

[0127] Step 2.2: Website design for data annotation. The present invention improves the efficiency and accuracy of data annotation by designing an online data annotation website. During the data annotation process, the administrator uploads the video samples to be annotated to the online website. The annotators can watch the videos directly on the website and annotate the behavior categories of the students in the video by single or multiple selection. The data annotation website mainly includes the following five parts: data table management, basic information management, automatic annotation of the student positions in the video, fine-tuning of the student positions in the video, and annotation of the student behaviors in the video.

[0128] (1) Database table design. The data annotation website uses MySQL database as a data management tool, which includes the following three data tables: user table, video table and annotation information table. The structural design of the relevant tables is as follows:

[0129] The user table is used to manage the relevant information of the user logging into the website, including the user's unique identifier (user_id), user name (user_name), email address (email) and login password (password). The data table structure design is shown in Table 1:

[0130] Table 1 User data table design

[0131] Field Name Field meaning Field constraints in MySQL user_id User unique identifier Integer, primary key, auto-increment user_name username Variable-length data type, maximum length 50, non-null email Mail Variable-length data type, maximum length 50, non-null password password Variable-length data type, maximum length 255, non-null

[0132] The video table is used to manage the video samples in the dataset. It contains the following fields: video unique identifier (video_id), video upload user (user_id), video name (title), and video path on the server (file_path). The data table structure design of the video table is shown in Table 2:

[0133] Table 2 Video data table design

[0134] Field Name Field meaning Field constraints in MySQL video_id Video unique identifier Integer, primary key, auto-increment user_id Video upload user ID Integer, foreign key (associated with user_id in the user table) title Video Name Variable-length data type, maximum length 100, non-null file_path The address of the video on the server Variable-length data type, maximum length 255, non-null

[0135] The annotation information table is used to record the specific position and behavior annotation information of each student appearing in the video. It contains the following fields: video unique identifier (video_id), student identifier in the video (person_id), the relative distance of each student from the video boundary (left boundary left, upper boundary top, right boundary right, lower boundary bottom) and the behavior of the person at that position (behaviors). Different behaviors are separated by commas. The data table structure design is shown in Table 3:

[0136] Table 3 Design of labeling information table

[0137]

[0138] (2) Basic information management mainly includes user management and video management.

[0139] User management: Users of this website are divided into two categories: administrators and ordinary users. Administrators can create new ordinary users, administrator users, modify basic user information, view the list of all users, upload videos, delete videos, view the list of all uploaded videos, etc. Ordinary users have the following three permissions: log in to the system, register an account, and annotate videos.

[0140] (3) Automatic labeling of students’ positions in videos: In order to improve the efficiency of labeling students’ spatial positions in videos, this website first calls the pre-trained YOLOv7 (You Only Live Once v7, referred to as YOLOv7) model to automatically identify all students’ position coordinates and confidence levels in each frame of the sample video, and sets the student detection box with a confidence level greater than 0.9 to represent a real student position.

[0141] (4) Fine-tune the position of students in the video. In view of the false detection and missed detection of the automatic annotation of student position information, this data annotation website has set up a manual modification module. The annotators manually review and upload the correct student position information.

[0142] (5) Annotation of students’ behaviors in the video. In the behavior category annotation module of the annotation website, annotators click to annotate 26 behaviors in a single-select or multiple-select manner. Finally, the real behavior labels are determined based on the annotation results of two annotators to ensure the scientificity and accuracy of the newly created data set.

[0143] Step 2.3: Dataset statistics. The statistics of the newly created dataset are shown in Table 4. During the creation of the dataset, a total of 66 videos of students listening to lectures were collected. From these original videos, 1906 2-second student video clips with frequent changes in student behavior were selected. A total of 54,461 student behavior annotations were included.

[0144] Table 4 Dataset statistics

[0145]

[0146] Step 3: Extraction and fusion of multiple features. The present invention extracts five features related to students' classroom behavior, namely, 3D joint angle features, 2D joint angle features of posture features, global body direction features, position offset features, and video appearance information, and designs a multi-feature fusion method based on deep learning.

[0147] Step 3.1: 3D joint angle feature extraction. Considering the impact of changes in students’ body parts in the video on behavior recognition, the PHALP (Predicting Human Appearance, Location and Pose for Tracking, referred to as PHALP) algorithm is used to extract the 3D joint angle information of the person in each frame of the video. i , where J i ∈R T×135 , T represents the length of the video, and i represents the i-th student appearing in the video.

[0148] Step 3.2: Posture feature extraction. The SMPL (Skinned Multi-Person Linear, SMPL for short) model is used to extract the student’s posture feature representation in each frame of the video. This includes the 2D joint angle feature θ i ∈R T×23×3×3 (23 human body joints are represented by 3×3 vectors respectively), global direction features of the body And the position offset feature ω i ∈R T×10 , where T represents the length of the video and i represents the i-th student appearing in the video.

[0149] Step 3.3: Video appearance feature extraction. Although human posture plays a key role in understanding actions, more complex behaviors require reasoning about scenes and contexts. Therefore, the present invention extracts video appearance features for each student in the video at the same time. The PHALP algorithm is used to track and obtain the trajectory of each student in the video, and the pre-trained 2D action recognition model MViT (Multiscale Vision Transformers, MViT for short) is run, and its feature vector before the classification layer is saved as the video appearance feature of each student in the video, i.e., A i ∈R T×4096 .

[0150] Step 3.4: Multi-feature fusion. For details, see Figure 2 As shown. Through steps 3.1, 3.2 and 3.3, multiple feature representations of students in the video can be obtained. Among them, the features extracted by SMPL in step 3.2 are linearly spliced ​​in the time dimension as the posture feature representation of the i-th student in the video. The posture features of N students in the video can be expressed as P = [P1,…,P i ,…P N ]∈R (N×T)×226 Furthermore, the feature of the i-th student in a video can be expressed as G i =[J i ,Pi ,A i ].

[0151] First, two linear networks are used to 3D joint angle information J i Encoding, the formula is:

[0152]

[0153] Among them, J i ∈R T×135 Represents the 3D joint angle information of the i-th student in the video, using the ReLU activation function; Encode network parameters for 3D joint angle information. Represents the encoded features of the 3D joint angle information of the i-th student.

[0154] Next, a linear network is used to train the pose feature P i Encoding, the formula is:

[0155]

[0156] Among them, P i ∈R T×226 Represents the posture information of the i-th student in the video; Encode network parameters for pose information; Represents the encoded features of the posture information of the i-th student.

[0157] Secondly, use the linear network and the bidirectional GRU network to analyze the appearance feature A i Encode, for the appearance feature A i ∈R T ×4096 Each time step feature x t ∈R 1×4096 Perform forward GRU calculation, the formula is:

[0158]

[0159]

[0160] Among them, σ(·) represents the Sigmoid activation function, ⊙ represents the matrix product, is the input weight matrix, is the weight matrix of the hidden state at the previous time step, are the parameters of GRU. Represents the hidden state of the feature at the tth moment in the forward GRU network.

[0161] For appearance feature A i ∈RT×4096 Each time step feature x t ∈R 1×4096 Perform reverse GRU calculation, the formula is:

[0162]

[0163] Among them, σ(·) represents the Sigmoid activation function, ⊙ represents the matrix product, is the input weight matrix, is the weight matrix of the hidden state at the previous time step, are the parameters of GRU. Represents the hidden state of the feature at the tth moment in the reverse GRU network.

[0164] At each time step t, the final hidden state h of the bidirectional GRU t is the forward hidden state and the reverse hidden state connection, that is Appearance Features A i The final output of the bidirectional gated recurrent neural network is h = [h1,h2…,h T ]∈R T×8192 .

[0165] Next, a linear network is used to encode the GRU output feature h of the appearance feature, and the formula is:

[0166]

[0167] in, Encode network parameters for appearance features; Represents the encoded features of the posture information of the i-th student.

[0168] Finally, the encoding features of the 3D joint angles Posture encoding features And appearance characteristics Perform linear splicing in the time dimension to obtain fusion features The formula is;

[0169]

[0170] in, represents feature concatenation, represents the fusion encoding features of the i-th student. T represents the time length of the video.

[0171] Step 3.5: Fusion of multi-person implicit relationship features. Usually, when predicting the behavior of the target student, information such as the behavior or body changes of other students is also very important. In order to use the information of other students in the video scene to better predict the behavior of the target student. Here, a multi-layer Transformer network structure is used to fuse the relationship features of this potential association. Multi-person implicit relationship feature fusion is mainly divided into the following two modules: location information extraction and Transformer fusion.

[0172] (1) Location information extraction. The YOLOv7 model is used to detect the location of students in each frame of the video, and the location information of N students in the video can be obtained. Locations = {L1, L2, ... L i ,…,L N}, where L i =[left i ,top i ,right i ,bottom i ]∈R 1×4 ,left i Indicates the relative pixel distance between the detected left border of the student box and the left border of the video. i Indicates the relative pixel distance between the detected upper border of the student box and the upper border of the video, right i Indicates the relative pixel distance between the right border of the detected student box and the right border of the video, bottom i Indicates the relative pixel distance between the detected lower border of the student box and the lower border of the video.

[0173] Next, the position encoding feature L i and student characteristics Perform the splicing operation, the formula is:

[0174]

[0175] in represents the feature representation of the i-th student in the video with position embedding added. Represents the concatenation of feature vectors in the time dimension.

[0176] (2) Transformer fusion. Given a student classroom video V, whose scene contains information about N students, through the above feature encoding steps, the video feature representation can be obtained Among them, F∈R N×T×1028 , transform the video feature representation F to satisfy the input of the Transformer network, that is, F∈R 1×(N×T)×1028 , the formula is:

[0177]

[0178] Among them, Transformers(·) represents a multi-layer Transformer structure, the number of layers is set to 12, and the number of hidden layer units is set to 512. Represents the result of N student features in a given video being encoded by the Transformers network.

[0179] Step 4: Build a multi-task decoder. Figure 3 In the decoding process, a pose estimation decoder and a video action recognition decoder are designed for pose estimation tasks and video action recognition tasks respectively.

[0180] Step 4.1 Pose Estimation Decoder. The pose estimation decoder contains three submodules, which respectively analyze the three pose features extracted by the SMPL algorithm (2D joint angle feature θ i , global orientation characteristics of the body And the position offset feature ω i ) for regression training.

[0181] For the 2D joint angle features θ of all students in the video, a multi-layer perceptron network is used for decoding. The formula is:

[0182]

[0183] Here, σ(·) represents the Sigmoid activation function. It represents the feature representation of the N initial features of students in the video after being encoded by the encoding layer. Decode network parameters for 2D joint angles; Represents the 2D joint angle decoded feature representation of N students in the video.

[0184] Global body orientation features for all students in the video Use a multi-layer perceptron network for decoding operations. The formula is:

[0185]

[0186] in, It represents the feature representation of the N initial features of students in the video after being encoded by the encoding layer. σ(·) represents the Sigmoid activation function. Decoding network for global body orientation; Decoded representation of the global body orientation features of N students in the video.

[0187] For the position offset features ω of all students in the video, a multi-layer perceptron network is used for decoding. The formula is:

[0188]

[0189] in, It represents the feature representation of the N initial features of students in the video after being encoded by the encoding layer. σ(·) represents the Sigmoid activation function. Decode network parameters for position offset; Decoded representation of the position offset features of N students in the video.

[0190] Finally, the three decoded features are concatenated in the second dimension to obtain the final posture feature decoding representation

[0191] Step 4.2 Video Behavior Classification Decoder. The video behavior classification decoder is mainly used for decoding behavior recognition. 26 binary classifiers are used to identify whether the target student has the corresponding 26 behaviors. The output of the model encoding layer for the nth and the student at the tth time in the video Perform behavior recognition decoding. The specific decoding formula for the student behavior that includes behavior category c is:

[0192]

[0193] in, represents the decoded representation of category c for the nth student in video V at time t. represents the fused feature representation of the nth student in video V at time t, 512 represents the dimension of the feature, and σ(·) represents the Sigmoid activation function. is the decoding network parameter of category c, where c represents one of the 26 student classroom behavior categories. Finally, the decoding of the behavior recognition decoder of the nth student at the tth moment in the video on the 26 behavior recognitions is expressed as

[0194]

[0195] The decoding results of the nth student in the video at the tth time on the 26 behaviors are normalized, and the formula is:

[0196]

[0197] in, It is the feature representation obtained by normalizing the decoded features on 26 behavior categories. n represents the nth student in the video, and t represents the tth moment in the video. represents the probability that the student’s decoding behavior contains behavior category c1, It represents the probability that the student's decoding behavior does not contain the behavior category c1.

[0198] The probability of the decoder of the nth student in the final video V at the tth moment in the 26 behavior categories is .

[0199] Step 4.3 Multi-task loss function setting. The overall target loss of the multi-task model is obtained by weighted addition of the losses of the two tasks of behavior recognition and posture estimation. Among them, the posture regression task uses the mean square error loss function to calculate its loss, and the behavior recognition task uses the cross entropy loss function to calculate its loss.

[0200] The pose estimation regression task uses the mean square error loss function to calculate the model loss. Specifically, the true pose feature of the nth student at time t in the video is represented by P n,t ∈R 1×226 , the feature representation of the model decoding of the nth student in the video at time t is n represents the nth student in the video, and t represents the tth moment in the video. The loss calculation formula for the pose estimation regression task is:

[0201]

[0202] Among them, L pose is the loss value of the pose estimation task of the nth student in the video at the tth time, P n,t ∈R 1×226 Represents the posture feature label of the nth student in the video at the tth moment. It represents the decoded features of the posture decoder of the nth student in the video at the tth moment.

[0203] The student behavior recognition task uses the cross entropy loss function to calculate the loss, as follows: The feature representation of the nth student in the video at the tth time is decoded by the model: The true label y of the nth student in the video at time t n,t ∈R 1×26 ,y n,t (i) is the vector y n,t The i-th element of :

[0204]

[0205] The loss calculation formula for a single student at time t is:

[0206]

[0207] Finally, the overall loss of the model is calculated as:

[0208] L=αL pose +(1-α)L action (twenty four)

[0209] L is the overall loss function value of the multi-task, and α is a hyperparameter used to balance the importance of the two tasks and is set to 0.3.

[0210] Step 5: Design evaluation indicators and experimental parameters.

[0211] Step 5.1 Evaluation index design. The present invention uses precision, recall and mean average precision mAP (mAP) to measure its performance on the data set. The calculation process of mAP value requires the use of two evaluation indicators, precision and recall, while the calculation of precision and recall requires the use of confusion matrix, which represents the correspondence between the true label and the predicted result, namely TP, FP, TN, and FN.

[0212] Among them, TP (True Positive, TP for short) represents the number of positive samples correctly identified, FN (False Negative, FN for short) represents the number of positive samples mistakenly identified as negative examples, FP (False Positive, FP for short) represents the number of negative samples mistakenly identified as positive examples, TP+FN represents the total number of positive samples, and TP+FP represents the total number of samples identified as positive examples.

[0213] Precision: The proportion of samples that are actually positive examples among all samples predicted to be positive examples (TP and FP). The formula is:

[0214]

[0215] Recall: The proportion of samples that are correctly predicted as positive examples among the samples whose true labels are positive examples (TP and FN). The formula is:

[0216]

[0217] Average Precision (AP) is an indicator used to evaluate the performance of a model in a specific category. First, for a specific category, the prediction results are sorted according to their confidence. Then, Precision and Recall are calculated based on the sorted detection results, and then the Precision-Recall curve is drawn. The area between the curve and the coordinate axis is the AP value of the category.

[0218] Under different confidence conditions, multiple sets of different Precision and Recall values ​​can be obtained. Among them, AP represents the average of all precision values ​​in the process of recall value changing from 0 to 1. The mean average precision (mAP) is the final indicator value obtained by averaging the AP values ​​of each category.

[0219] Mean Average Precision (mAP): The average AP of each category is taken as the final model mAP indicator value, and its formula is:

[0220]

[0221] Where N represents the number of student behavior categories, which is set to 26. i Represents the average precision value of the i-th behavior category.

[0222] The experimental indicators used in the present invention are average precision (the average of the precision values ​​of all categories), average recall (the average of the recall values ​​of all categories), and average precision mean (the average of the average precision of all categories).

[0223] Step 5.2 Experimental parameter design:

[0224] The batch size is set to 4. A too small batch size will slow down the convergence of the model during training and affect the training efficiency. However, increasing the batch size appropriately can improve the generalization ability of the model and accelerate the training process. The initial value of the learning rate is set to 0.0008, the number of iterations is set to 10, and the learning rate decay rate is set to 0.1. The hidden layer dimension in the encoding layer is set to 512 and the number of Transformer layers used is set to 12. The multi-task weight hyperparameter α is set to 0.3. The dropout rate of the linear network layer of the multi-layer perceptron network is set to 0.3.

[0225] Step 6: Design a comparative experiment, collect statistics on the experimental results and analyze the experimental results to test the usefulness of the method proposed in the present invention.

[0226] Step 6.1: Select a baseline model for comparison.

[0227] (1) Convolutional 3D (C3D) is a deep learning model for video analysis that can effectively capture the spatiotemporal features in videos.

[0228] (2) 3D residual convolutional network (ResNet3D, abbreviated as R3D) is a deep residual network based on 3D convolution, which can more effectively capture the spatiotemporal features in videos, solve the difficulties in deep network training, and effectively alleviate the problems of gradient vanishing and gradient exploding.

[0229] (3) Shifted Window Transformer (Swin) is an efficient window-based Transformer model. Its attention mechanism can capture global information across windows, rather than just being limited to the local window. It not only retains the powerful feature extraction capability of the Transformer model, but also solves the computational bottleneck problem of high-resolution image processing.

[0230] (4) Unified Transformer (Uniformer) is a deep learning model used in computer vision tasks. It combines convolutional layers and Transformer structures to improve the richness of feature representation while maintaining high efficiency.

[0231] (5) The Actor-Context-Actor Relation (ACAR) network not only models the direct relationships between entity pairs in the video, but also considers the indirect high-order relationships built on multiple elements.

[0232] (6) The dual-branch convolutional network (SlowFast Network, referred to as SlowFast) can more effectively capture the dynamic and static features in the video by processing the fast and slow information in the video in parallel. Among them, the Slow branch focuses on capturing the static and slow-changing features in the video, while the Fast branch focuses on capturing the fast-moving and dynamic-changing features in the video.

[0233] (7) Video Masked Autoencoder V2 (VideoMAE V2) is a self-supervised learning method for video understanding. It improves on the original VideoMAE and further improves the performance of video tasks through more effective pre-training and architecture design.

[0234] (8) Hierarchical Vision Transformer (Hiera): Hiera is a new neural network architecture designed to efficiently process image and video data. This architecture combines the ideas of hierarchical feature representation and multi-scale information processing, and has high computational efficiency and superior performance.

[0235] (9) Spatio-Temporal Aggregation and Relation Network++ (STAR++) is an advanced network architecture for video understanding and analysis. It captures dynamic and static information in videos by effectively integrating spatiotemporal features.

[0236] Step 6.2: Conduct a comparative experiment between the model proposed in this invention and the baseline model.

[0237] The model proposed in the present invention is compared with the baseline model. The experimental results are shown in Table 5, where Ours is a fine-grained student classroom behavior recognition method assisted by posture features and regression tasks proposed in the present invention. The model proposed in the present invention is superior to other baseline models in all indicators. The model can effectively integrate 3D joint angle features, posture features and video appearance features, and the average accuracy of all student behavior categories reaches 0.553, which proves the effectiveness of students' posture information in the task of student classroom behavior recognition.

[0238] Table 5 compares the experimental results. The best results are shown in bold.

[0239]

[0240] Step 6.3: Multi-task training analysis.

[0241] To verify the effectiveness of multi-task training, the present invention conducted multi-task training experiments and single-task training experiments. First, in the decoder part, the posture regression task and the behavior prediction task are performed simultaneously. This experimental scheme is named Multi-task Network. Second, in the decoder part, the posture regression task is frozen and only the video behavior classification task is performed. This experimental scheme is named Single-task Network. The specific experimental results are shown in Table 6:

[0242] Table 6 Multi-task experiment analysis

[0243]

[0244] According to the experimental results in Table 6, compared with the single-task training architecture, the average accuracy of action recognition in the multi-task training experiment has been improved. From this, we can draw the following conclusion: the posture regression task has certain benefits for the action recognition task.

[0245] Step 6.4 Multi-task loss hyperparameter analysis

[0246] In order to select the best task weights for the posture regression task and the video behavior classification task, the hyperparameter α is adjusted. The experimental results are shown in Table 7.

[0247] Table 7 Experimental analysis of the effect of hyperparameter α on experimental performance

[0248]

[0249] From the above experimental results, it can be found that when the hyperparameter α = 0.3, the average precision of the experiment is the highest and the model performance is the best. Therefore, α = 0.3 is finally selected as the optimal model parameter. This may be because students may not perform many large movements in class, so a lower task weight is given to the posture regression task.

[0250] Embodiment 1:

[0251] This embodiment relates to a student classroom behavior recognition system based on video analysis, which can monitor and analyze students' classroom behaviors in real time and generate corresponding behaviors. The system includes the following modules:

[0252] (1) Video acquisition module: Multiple high-definition cameras are installed in the classroom to collect real-time video data of students’ classroom behavior. The cameras are placed in different locations in the classroom to ensure that all students in the classroom are covered.

[0253] (2) Feature extraction module: Use the YOLOv7 model to detect the positions of all students in the video. Use the PHALP algorithm to extract the 3D joint angle information of the person in each frame of the video, and use the SMPL model to extract the posture feature representation of the person in each frame of the video. These include 2D joint angle features, global body direction features, and position offset features. Use the PHALP algorithm to track and obtain the trajectory of each student in the video, run the pre-trained 2D action recognition model MViT model, and save its feature vector before the classification layer as the video context appearance feature of each person in the video.

[0254] (3) Behavior recognition module: First, the collected video is preprocessed by denoising and image enhancement to improve the video quality. Figure 4 As shown, the model trained by the present invention is used to track the trajectory and identify the behavior of each student in the video.

[0255] The experimental results of the embodiment are shown in Table 8:

[0256] Table 8 System test sample results

[0257]

[0258] In the example test, the posture features of Student 1 effectively captured his head state and arm movements, and the final recognition result was consistent with the true behavior label, demonstrating the importance of posture features in student behavior recognition.

[0259] In the example test, for Student 2, the system was able to identify body movements such as "raising head", "standing straight", and "looking forward", but failed to identify the action of "holding a pen in hand". The model failed to capture the subtle pen-related features in the video. In addition, observing the original video frame, it was found that the student was blocked by a bottle of water in front of him, which caused the model to have certain problems in dealing with occlusion problems and identifying small objects interacting with students, which needs to be optimized and further studied.

[0260] In the example test, for Student 3, the system successfully identified behaviors such as "lowering head", "leaning left", "looking at the phone", and "holding the phone in hand". For the behavior of "holding the phone in hand", the model can mention the interaction between the hand and the phone through video features, and the system can also accurately identify this type of behavior.

[0261] Embodiment 2:

[0262] In order to more comprehensively verify the benefits of the model proposed in this invention, detailed comparative experiments were conducted on multiple datasets of different scales and scenarios, including the public datasets UCF101-24 and AVAv2.1.

[0263] The public dataset UCF101-24 is a subset of the action recognition dataset UCF101 (University of CentralFlorida 101), which contains 24 action categories, including sports activities or daily behaviors such as "basketball shooting", "playing guitar", and "playing golf". The dataset contains a total of 3207 video samples.

[0264] The public dataset AVAv2.1 (Atomic Visual Actions v2.1, AVAv2,1) is a video dataset for action recognition based on movie scenes. The dataset is annotated with 80 different action categories. These action categories cover simple actions (such as "standing" or "walking") to more complex interactions (such as "making a phone call" or "shaking hands"). The dataset contains a total of 211,236 annotated video clips.

[0265] This example selects the top-3 algorithms in behavior recognition performance on the public datasets UCF101-24 and AVAv2.1 as comparison models. The comparison models include STAR, YOWO, HIT, ACAR, and JMRN.

[0266] STAR (SkeleTal Action Recognition, STAR) processes the spatiotemporal information composed of key points of the human skeleton through deep learning methods to recognize and classify different actions.

[0267] YOWO (You Only Watch Once) aims to recognize behaviors by processing a sequence of video frames in a single pass. It combines convolutional neural networks and long short-term memory networks to capture the spatiotemporal characteristics of videos.

[0268] HIT (Human Interaction Transformer, HIT) uses Transformer to capture long-distance dependencies and complex spatiotemporal features in video data, and can effectively handle the dynamic changes of character actions and interactions in videos.

[0269] ACAR (Actor-Context-Actor Relation, ACAR) models the interactive relationship between two or more participants in a video, while taking into account their interaction with the environment, which can better understand complex scenes and interactive behaviors.

[0270] JMRN (Joint Motion Representation Network) provides a comprehensive action description by integrating various feature representations, thereby improving the recognition ability of complex actions.

[0271] The comparative experimental results of the proposed method on the public datasets UCF101-24 and AVAv2.1 are shown in Table 9:

[0272] Table 9 Experimental results of public datasets UCF101-24 and AVA v2.1 in different scenarios and scales

[0273]

[0274]

[0275] The following analysis can be obtained from the experimental results in Table 9:

[0276] (1) On the UCF101-24 dataset, the proposed method performs well, with an average precision of 0.890, second only to the current best performing algorithm STAR / L (0.903). Compared with other recent outstanding algorithms, such as YOWO+LFB (0.873) and HIT (0.848), the proposed method shows strong competitiveness. However, the video samples in the UCF101-24 dataset mostly focus on the actions of a single person, and rarely involve complex multi-person interaction scenes. This lack of information on behavioral interactions between people may limit the ability of the proposed method to capture rich spatiotemporal features. Despite this, the proposed method still achieves near-optimal recognition accuracy by effectively extracting the spatiotemporal features of a single person's actions. This result demonstrates the beneficial effects of the proposed method.

[0277] (2) On the AVA v2.1 dataset, the proposed method achieved an average precision of 0.421, slightly exceeding the current best performing algorithm STAR / L (0.417). This result shows that the proposed method performs well in handling complex action recognition tasks, especially in cases involving multi-person interactions and rich scene contexts. This excellent performance can be attributed to the innovation of the proposed method in extracting and fusion of multiple features, which can fully extract and utilize features related to action recognition.

[0278] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made to the present invention shall be included in the protection scope of the present invention.

Claims

1. A fine-grained student classroom behavior recognition method assisted by posture features and regression tasks, characterized by: The following steps are involved: Step 1: Define student classroom behaviors: When selecting student classroom behavior categories, 26 types of student classroom behaviors were defined based on student body changes, including head, body, sight, hands, and mouth, including 7 head behaviors, 4 body behaviors, 5 sight behaviors, 9 hand behaviors, and 1 mouth behavior; Step 2: Establish a website and method for annotating students’ classroom behaviors, and construct a dataset of students’ classroom video behaviors; design the criteria for dataset annotation and the website for annotating video data based on the definition of students’ classroom behaviors and the collected students’ classroom videos; Step 3: Extraction and fusion of multiple features: Extract five features related to students’ classroom behavior, namely 3D joint angle features, 2D joint angle features of posture features, global body direction features, position offset features, and video appearance information, and design a multi-feature fusion method based on deep learning; Step 4: Build a multi-task decoder: During the decoding process, design a pose estimation decoder and a video behavior recognition decoder for the pose estimation task and the video behavior recognition task, respectively.

2. The posture feature and regression task-assisted fine-grained student classroom behavior recognition method according to claim 1 is characterized by: The extraction and fusion methods of the multiple features described in step 3 are as follows: Step 3.1: 3D joint angle feature extraction, using the PHALP algorithm to extract the 3D joint angle information of the person in each frame of the video i , where J i ∈R T×135 , T represents the length of the video, i represents the i-th student appearing in the video; Step 3.2: Posture feature extraction: Use the SMPL model to extract the student's posture feature representation in each frame of the video; This includes the 2D joint angle feature θ i ∈R T×23×3×3 , 23 human body joints are represented by 3×3 vectors, and the global direction features of the body are And the position offset feature ω i ∈R T×10 , where T represents the length of the video, and i represents the i-th student appearing in the video; Step 3.3: Video appearance feature extraction, use the PHALP algorithm to track and obtain the trajectory of each student in the video, run the pre-trained 2D action recognition model MViT, and save its feature vector before the classification layer as the video appearance feature of each student in the video, that is, A i ∈R T×4096 ; Step 3.4: Multi-feature fusion, through steps 3.1 to 3.3, obtain multiple feature representations of students in the video; The features extracted using SMPL in step 3.2 are linearly concatenated in the time dimension as the posture feature representation of the i-th student in the video. The posture features of N students in the video are expressed as P = [P1,…,P i ,…P N ]∈R (N×T)×226 ; Furthermore, the feature of the i-th student in a video is represented by G i =[J i ,P i ,A i ]; First, two linear networks are used to 3D joint angle information J i Encoding, the formula is: Among them, J i ∈R T×135 Represents the 3D joint angle information of the i-th student in the video, using the ReLU activation function; Encode network parameters for 3D joint angle information; The encoding features representing the 3D joint angle information of the i-th student; Next, a linear network is used to train the pose feature P i Encoding, the formula is: Among them, P i ∈R T×226 Represents the posture information of the i-th student in the video; Encode network parameters for pose information; The encoding features representing the posture information of the i-th student; Secondly, use the linear network and the bidirectional GRU network to analyze the appearance feature A i Encode, for the appearance feature A i ∈R T×4096 Each time step feature x t ∈R 1×4096 Perform forward GRU calculation, the formula is: For appearance feature A i ∈R T×4096 Each time step feature x t ∈R 1×4096 Perform reverse GRU calculation, the formula is: Among them, σ(·) represents the Sigmoid activation function, ⊙ represents the matrix product, is the input weight matrix, is the weight matrix of the hidden state at the previous time step, are the parameters of GRU; Represents the hidden state of the feature at the tth moment of the reverse GRU network; At each time step t, the final hidden state h of the bidirectional GRU t is the forward hidden state and the reverse hidden state connection, that is Appearance Features A i The final output of the bidirectional gated recurrent neural network is h = [h1,h2…,h T ]∈R T×8192 ; Next, a linear network is used to encode the GRU output feature h of the appearance feature, and the formula is: in, Encode network parameters for appearance features; The encoding features representing the posture information of the i-th student; Finally, the encoding features of the 3D joint angles Posture encoding features And appearance characteristics Perform linear splicing in the time dimension to obtain fusion features The formula is; in, represents feature concatenation, represents the fusion coding features of the i-th student; T represents the time length of the video; Step 3.5: Fusion of multi-person implicit relationship features: Use a multi-layer Transformer network structure to fuse the relationship features of potential associations; multi-person implicit relationship feature fusion is mainly divided into the following two modules: location information extraction and Transformer fusion.

3. The posture feature and regression task-assisted fine-grained student classroom behavior recognition method according to claim 2 is characterized by: The position information extraction uses the YOLOv7 model to detect the student position in each frame of the video, and obtains the position information of N students in the video. Locations = {L1, L2, ... L i ,…,L N }, where L i =[left i ,top i ,right i ,bottom i ]∈R 1×4 ,left i Indicates the relative pixel distance between the detected left border of the student box and the left border of the video. i Indicates the relative pixel distance between the detected upper border of the student box and the upper border of the video, right i Indicates the relative pixel distance between the right border of the detected student box and the right border of the video, bottom i Indicates the relative pixel distance between the detected lower border of the student box and the lower border of the video; Next, the position encoding feature L i and student characteristics Perform the splicing operation, the formula is: in represents the feature representation of the i-th student in the video with position embedding; Represents the concatenation of feature vectors in the time dimension.

4. The posture feature and regression task-assisted fine-grained student classroom behavior recognition method according to claim 2 is characterized by: The Transformer fusion is as follows: given a student classroom video V, whose scene contains information about N students, the video feature representation is obtained through the feature encoding step Among them, F∈R N×T×1028 , transform the video feature representation F to satisfy the input of the Transformer network, that is, F∈R 1×(N×T)×1028 , the formula is: Among them, Transformers(·) represents a multi-layer Transformer structure, the number of layers is set to 12, and the number of hidden layer units is set to 512; Represents the result of N student features in a given video being encoded by the Transformers network.

5. The posture feature and regression task-assisted fine-grained student classroom behavior recognition method according to claim 1 is characterized by: The specific steps of constructing the multi-task decoder described in step 4 are as follows: Step 4.1 Pose Estimation Decoder. The pose estimation decoder contains three submodules, which respectively extract the three pose features of SMPL algorithm, 2D joint angle feature θ i , global orientation characteristics of the body And the position offset feature ω i Conduct regression training; For the 2D joint angle features θ of all students in the video, a multi-layer perceptron network is used for decoding; the formula is: Where σ(·) represents the Sigmoid activation function; It represents the feature representation of the N initial features of students in the video after being encoded by the encoding layer; Decode network parameters for 2D joint angles; Represents the 2D joint angle decoded feature representation of N students in the video; Global body orientation features for all students in the video A multi-layer perceptron network is used for decoding; its formula is: in, represents the feature representation of the N initial features of students in the video after being encoded by the encoding layer; σ(·) represents the Sigmoid activation function; Decoding network parameters for global body orientation; Decoded representation of the global orientation features of the bodies of N students in the video; For the position offset features ω of all students in the video, a multi-layer perceptron network is used for decoding; the formula is: in, represents the feature representation of the N initial features of students in the video after being encoded by the encoding layer; σ(·) represents the Sigmoid activation function; Decode network parameters for position offset; Decoded representation of the position offset features of N students in the video; Finally, the three decoded features are concatenated in the second dimension to obtain the final posture feature decoding representation Step 4.2 Video behavior classification decoder; Video behavior classification decoder is used for decoding behavior recognition; 26 binary classifiers are used to identify whether the target student contains the corresponding 26 behaviors; the output of the model encoding layer for the nth time in the video and the student at the tth time Perform behavior recognition decoding. The specific decoding formula for the student behavior that includes behavior category c is: in, represents the decoded representation of the nth student in the video V at the tth time on the category c; represents the fused feature representation of the nth student in video V at time t, 512 represents the dimension of the feature; σ(·) represents the Sigmoid activation function; is the decoding network parameter of category c, where c represents one of the 26 student classroom behavior categories; finally, the decoding of the behavior recognition decoder of the nth student at the tth moment in the video on the 26 behavior recognitions is expressed as The decoding results of the nth student in the video at the tth time on the 26 behaviors are normalized, and the formula is: in, It is the feature representation obtained by normalizing the decoded features on 26 behavior categories, n represents the nth student in the video, and t represents the tth moment in the video; represents the probability that the student’s decoding behavior contains behavior category c1, represents the probability that the student's decoding behavior does not include the behavior category c1; the probability of the nth student's decoder in the 26 behavior categories at the tth moment in the final video V is, Step 4.3 Multi-task loss function setting: the overall target loss of the multi-task model is obtained by weighted addition of the losses of the two tasks of behavior recognition and posture estimation; among them, the posture estimation regression task uses the mean square error loss function to calculate its loss, and the behavior recognition task uses the cross entropy loss function to calculate its loss.

6. The posture feature and regression task-assisted fine-grained student classroom behavior recognition method according to claim 5 is characterized by: The pose estimation regression task described in step 4.3 uses the mean square error loss function to calculate the model loss; specifically, the true pose feature of the nth student in the video at time t is represented by P n,t ∈R 1×226 , the feature representation of the model decoding of the nth student in the video at time t is n represents the nth student in the video, t represents the tth moment in the video; the loss calculation formula for the posture estimation regression task is: Among them, L pose is the loss value of the pose estimation task of the nth student in the video at the tth time, P n,t ∈R 1×226 represents the posture feature label of the nth student in the video at the tth moment; It represents the decoded features of the posture decoder of the nth student in the video at the tth moment; The student behavior recognition task uses the cross entropy loss function to calculate the loss, as follows: The feature representation of the nth student in the video at the tth time is decoded by the model: The true label y of the nth student in the video at time t n,t ∈R 1×26 ,y n,t (i) is the vector y n,t The i-th element of : The loss calculation formula for a single student at time t is: Finally, the overall loss of the model is calculated as: L=αL pose +(1-α)L action (24) L is the overall loss function value of the multi-task, and α is a hyperparameter used to balance the importance of the two tasks and is set to 0.

3.

7. The posture feature and regression task-assisted fine-grained student classroom behavior recognition method according to claim 1 is characterized by: It also includes step 5: designing evaluation indicators and experimental parameters to implement multi-task model training; Step 6: Design a comparative experiment to test the usefulness of the method proposed in the present invention.

Images