Video human body behavior recognition method and system and electronic equipment

Abstract

The invention belongs to the technical field of computer vision, particularly relates to a video human behavior recognition method and system and electronic equipment, and aims to solve the problem that behavior recognition cannot meet the requirements of low calculation cost and high recognition precision at the same time. The method comprises the following steps: performing human body tracking on a video to obtain a time sequence track of a target human body; performing time cutting and space clipping on the video; obtaining a time-space cutting short time sequence video; and a Video MAE model based on a visual Transform is used to identify the action of the target human body in the time-space cut short time sequence video to obtain an action time sequence of the target human body, and the Video MAE model comprises at least four Transform layers. According to the video human body behavior identification method and system provided by the invention, on the premise of ensuring the identification precision, the calculation amount can be greatly reduced, the method is adaptive to an environment with limited calculation power, and the behavior identification efficiency is improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a video human behavior recognition method, system, electronic device and computer-readable storage medium, as well as a computer program product containing instructions. Background Technology

[0002] Currently, video-based behavior recognition technology is gradually becoming a research hotspot. By extracting fine-grained motion features of the human body through computer vision and deep learning techniques, it can provide objective and traceable data support for human behavior analysis and evaluation, effectively compensating for the shortcomings of traditional manual analysis. In the field of video understanding technology, deep learning models have made breakthrough progress. Among them, the Vision Transformer (ViT) model, with its ability to model global feature dependencies, has demonstrated superior performance to traditional convolutional neural networks in tasks such as action recognition and behavior analysis, providing a technological foundation for video human behavior recognition.

[0003] However, existing video action recognition technology still faces many problems that urgently need to be solved in practical applications:

[0004] First, there is a significant contradiction between computational complexity and deployment adaptability. Existing ViT models and video understanding models based on them typically require a large number of parameter calculations and high computing power to ensure the integrity of feature extraction, resulting in slow model inference speed and high resource consumption.

[0005] Secondly, general models are not well adapted to certain scenarios, such as education. Teaching videos in education scenarios have distinct domain specificity: on the one hand, the videos contain multiple target subjects such as teachers and students; on the other hand, the core classroom behaviors are mostly fine-grained small actions (such as raising hands, taking notes, eye contact, etc.), with small movement amplitude and low feature discrimination. Video recognition models designed for general scenarios are difficult to accurately capture the unique behavioral features of education scenarios, resulting in low action recognition accuracy and high misjudgment rate.

[0006] Furthermore, the existing system lacks modularity and scalability. When it is necessary to adapt to different scenarios or add new behavioral evaluation indicators, the system needs to be reconstructed on a large scale. It cannot achieve flexible reuse and rapid expansion of modules, which increases the cost of technology iteration and scenario adaptation.

[0007] In summary, existing technologies cannot simultaneously meet the practical requirements of "low computational cost, high recognition accuracy, convenient deployment, and strong scene adaptability." Therefore, how to reduce computational cost while ensuring action recognition accuracy and possess modular scalability is an urgent problem to be solved. Summary of the Invention

[0008] To address the aforementioned technical problems in the prior art, namely how to reduce computational costs while ensuring action recognition accuracy and possessing modular scalability, this application provides a video human behavior recognition method and system.

[0009] In a first aspect of this application, a video human behavior recognition method is provided, comprising:

[0010] Human body tracking is performed on the video to obtain the temporal trajectory of the target human body. The temporal trajectory includes the coordinates of the bounding box of the target human body, the video frame number, and the trajectory ID.

[0011] The video is segmented according to a preset duration to obtain continuous short time-series videos and short time-series trajectories corresponding to the short time-series videos;

[0012] Based on the width and height of the bounding box of each target human body in each frame of the short temporal video, and according to the preset human upper body ratio and environment expansion ratio, the reserved area corresponding to each target human body in each frame is obtained, and the area outside the reserved area corresponding to each target human body in each frame is cropped to obtain the spatiotemporal segmented short temporal video.

[0013] The VideoMAE model, based on a visual Transformer, is used to identify the actions of the target human body in the spatiotemporally segmented short temporal video, thereby obtaining the temporal sequence of the target human body's actions. The VideoMAE model includes C Transformer layers, where C ≥ 4.

[0014] Optionally, the step of using the VideoMAE model based on the visual Transformer to identify the actions of the target human body in the spatiotemporally segmented short-sequence video, and obtaining the action time sequence of the target human body, includes:

[0015] For each frame of the spatiotemporally segmented short temporal video, based on the token set of each frame... , collect tokens Tokens in odd positions are grouped into a token set TA1. The tokens in even positions of the list are grouped into a token set TA2. The feature similarity between the tokens in set TA1 and the tokens in set TA2 is calculated. Where S(i,j) is the feature similarity between the i-th token in set TA1 and the j-th token in set TA2, 1≤i≤n, 1≤j≤n;

[0016] For each token, calculate the feature similarity count of the token. The feature similarity count is the number of feature similarities S(i,j) of the token that are greater than a preset first threshold. If the feature similarity count is greater than a preset second threshold, add the token corresponding to the feature similarity count to the pre-pruning token set R of the corresponding frame, and sort the tokens in the pre-pruning token set R in descending order of the corresponding feature similarity count.

[0017] The final number of tokens to be pruned in each frame is calculated by averaging the number of tokens in the pre-pruned token set R for all frames.

[0018] For each frame, remove the first r tokens from the pre-pruning token set R, and generate a pruned token set from the remaining tokens. Where m ≤ n;

[0019] The token set of each frame of the spatiotemporally segmented short-sequence video. Input the first Transformer layer of the VideoMAE model to obtain the set of pruned tokens for each frame of the spatiotemporally segmented short temporal video. Input the second to the Cth Transformer layers of the VideoMAE model to identify the actions of the target human body in the spatiotemporally segmented short time-series video, and obtain the action time sequence of the target human body, where C is the number of the Transformer layers of the VideoMAE model.

[0020] Optionally, the step of using the VideoMAE model based on the visual Transformer to identify the actions of the target human body in the spatiotemporally segmented short-sequence video further includes:

[0021] The set of tokens for each frame of the h-th Transformer layer input to the VideoMAE model. , collect tokens Tokens in odd positions in the sequence are grouped into a token set. , collect tokens Tokens in even positions in the sequence are grouped into a token set. Calculate the set tokens and sets Feature similarity between tokens Where S(i,j) is a set The i-th token and the set The feature similarity between the j-th tokens is 1≤i≤k, 1≤j≤k; where k≤m, 4≤h≤C, and C is the number of Transformer layers.

[0022] For each frame, all tokens are sorted in descending order of their corresponding feature similarity S(i,j) to generate a sequence of similar token pairs. The first r highly similar token pairs in the sequence are then weighted and averaged according to their token size to generate a fused token. ,in and They are tokens and token Size, token and token Belongs to the highly similar token pair, For the fusion token, The size is ;

[0023] For the fusion token, the Transformer layer attention A Where Q is the key vector of the fusion token, K is the value vector of the fusion token, d is the vector dimension of the fusion token, and s is the size of the fusion token;

[0024] Replace the token set with the fused token. The corresponding token pairs in the set yield a fusion token set. ;

[0025] The fusion token set Input the multilayer perceptron submodule of the h-th Transformer layer to identify the actions of the target human body in the spatiotemporally segmented short time-series video and obtain the action time sequence of the target human body.

[0026] Optionally, the step of performing human body tracking on the video to obtain the temporal trajectory of the target human body includes:

[0027] The ByteTrack algorithm is used to perform human body tracking on the video.

[0028] In the detection phase of the ByteTrack algorithm, the YOLOX-X model based on the Anchor-free architecture is used, with a confidence threshold of 0.5.

[0029] In the association phase of the ByteTrack algorithm, the detection result of the current frame in the detection phase is matched with the detection result of the previous frame, including:

[0030] Calculate the intersection-union ratio (IUGR) of each trajectory in the current detection result with each trajectory in the previous frame detection result. If the IUGR is greater than or equal to a preset third threshold, determine that the trajectory in the current detection result and the trajectory in the previous frame detection result corresponding to the IUGR belong to the same target human body.

[0031] For each low-confidence trajectory in the current frame detection result whose confidence is less than the confidence threshold, if the intersection-over-union ratio (IoU) corresponding to the low-confidence trajectory is less than the third threshold, the current frame is determined to be a mismatched frame of the low-confidence trajectory, and the mismatched frame count of the low-confidence trajectory is incremented by 1.

[0032] If the number of mismatched frames of the low-confidence trajectory is greater than or equal to a preset fourth threshold, the low-confidence trajectory will be removed.

[0033] Optionally, the step of obtaining the reserved region corresponding to each target human body in each frame based on the width and height of the bounding box of each target human body in each frame of the short time-series video, according to a preset human upper body ratio and environment expansion ratio, includes: calculating the height of the reserved region. ; and calculate the width of the reserved area. Where H is the height of the bounding box, W is the width of the bounding box, upperbody_ratio is the proportion of the upper body, and environment_ratio is the environment expansion ratio.

[0034] Optionally, the video human behavior recognition method further includes:

[0035] Based on the action sequence of the target human body, calculate the behavioral evaluation index of the target human body and determine the behavioral evaluation level of the target human body.

[0036] Optionally, the video is an instructional video, the target human body is a student, the behavioral assessment index of the target human body is student behavioral engagement, the behavioral evaluation level of the target human body is learning engagement level, the action sequence of the target human body includes the sequence of various student behaviors, the various student behaviors include various behaviors that deviate from learning, and the step of calculating the behavioral assessment index of the target human body and determining the behavioral evaluation level of the target human body based on the action sequence of the target human body includes:

[0037] The total class time is obtained based on the duration of the video.

[0038] The duration and frequency of the student's disengagement behavior are determined based on the action sequence of the target human body.

[0039] Calculate students' original learning behavior input ,in, Indicates the total class time. Indicates the first The duration of the detachment from learning behavior, Indicates the first The number of times a disengagement behavior occurs, where M represents the total number of categories of disengagement behaviors;

[0040] Calculate the student's behavioral engagement ,in, The average initial learning engagement of all students. , Let N be the initial learning engagement level of the r-th student, and N be the number of students. The standard deviation of the distribution of all students' initial learning engagement is represented by the following: ;

[0041] If the student's behavioral engagement is greater than or equal to a preset fifth threshold, the learning engagement level is determined to be a high engagement level; if the student's behavioral engagement is less than the fifth threshold, the learning engagement level is determined to be a low engagement level.

[0042] In a second aspect of this application, a video human behavior recognition system is provided, the system comprising:

[0043] The human body tracking module is used to track human bodies in videos and obtain the temporal trajectory of the target human body. The temporal trajectory includes the coordinates of the bounding box of the target human body, the video frame number, and the trajectory ID.

[0044] The time segmentation module is used to divide the video into segments according to a preset duration to obtain continuous short time-series videos and short time-series trajectories corresponding to the short time-series videos;

[0045] The spatial cropping module is used to obtain the reserved area corresponding to each target human body in each frame according to the width and height of the bounding box of each target human body in each frame of the short temporal video, according to the preset human upper body ratio and environment expansion ratio, and to crop and remove the area outside the reserved area corresponding to each target human body in each frame, so as to obtain the spatiotemporally cut short temporal video.

[0046] The action recognition module is used to identify the actions of the target human body in the spatiotemporally segmented short time-series video using the VideoMAE model based on visual Transformer, and to obtain the action time sequence of the target human body. The VideoMAE model includes C Transformer layers, where C≥4.

[0047] Optionally, the action recognition module is specifically used for:

[0048] For each frame of the spatiotemporally segmented short temporal video, based on the token set of each frame... , collect tokens Tokens in odd positions are grouped into a token set TA1. The tokens in even positions of the list are grouped into a token set TA2. The feature similarity between the tokens in set TA1 and the tokens in set TA2 is calculated. Where S(i,j) is the feature similarity between the i-th token in set TA1 and the j-th token in set TA2, 1≤i≤n, 1≤j≤n;

[0049] For each token, calculate the feature similarity count of the token. The feature similarity count is the number of feature similarities S(i,j) of the token that are greater than a preset first threshold. If the feature similarity count is greater than a preset second threshold, add the token corresponding to the feature similarity count to the pre-pruning token set R of the corresponding frame, and sort the tokens in the pre-pruning token set R in descending order of the corresponding feature similarity count.

[0050] The final number of tokens to be pruned in each frame is calculated by averaging the number of tokens in the pre-pruned token set R for all frames.

[0051] For each frame, remove the first r tokens from the pre-pruning token set R, and generate a pruned token set from the remaining tokens. Where m ≤ n;

[0052] The token set of each frame of the spatiotemporally segmented short-sequence video. Input the first Transformer layer of the VideoMAE model to obtain the set of pruned tokens for each frame of the spatiotemporally segmented short temporal video. Input the second to the Cth Transformer layers of the VideoMAE model to identify the actions of the target human body in the spatiotemporally segmented short time-series video, and obtain the action time sequence of the target human body, where C is the number of the Transformer layers of the VideoMAE model.

[0053] Optionally, the action recognition module is further configured to:

[0054] The set of tokens for each frame of the h-th Transformer layer input to the VideoMAE model. , collect tokens Tokens in odd positions in the sequence are grouped into a token set. , collect tokens Tokens in even positions in the sequence are grouped into a token set. Calculate the set tokens and sets Feature similarity between tokens Where S(i,j) is a set The i-th token and the set The feature similarity between the j-th tokens is 1≤i≤k, 1≤j≤k; where k≤m, 4≤h≤C, and C is the number of Transformer layers.

[0055] For each frame, all tokens are sorted in descending order of their corresponding feature similarity S(i,j) to generate a sequence of similar token pairs. The first r highly similar token pairs in the sequence are then weighted and averaged according to their token size to generate a fused token. ,in and They are tokens and token Size, token and token Belongs to the highly similar token pair, For the fusion token, The size is ;

[0056] For the fusion token, the Transformer layer attention A Where Q is the key vector of the fusion token, K is the value vector of the fusion token, d is the vector dimension of the fusion token, and s is the size of the fusion token;

[0057] Replace the token set with the fused token. The corresponding token pairs in the set yield a fusion token set. ;

[0058] The fusion token set Input the multilayer perceptron submodule of the h-th Transformer layer to identify the actions of the target human body in the spatiotemporally segmented short time-series video and obtain the action time sequence of the target human body.

[0059] Optionally, the human body tracking module is specifically used for:

[0060] The ByteTrack algorithm is used to perform human body tracking on the video.

[0061] In the detection phase of the ByteTrack algorithm, the YOLOX-X model based on the Anchor-free architecture is used, with a confidence threshold of 0.5.

[0062] In the association phase of the ByteTrack algorithm, the detection result of the current frame in the detection phase is matched with the detection result of the previous frame;

[0063] When the human body tracking module matches the current frame detection result with the previous frame detection result during the detection phase, it is specifically used for:

[0064] Calculate the intersection-union ratio (IUGR) of each trajectory in the current detection result with each trajectory in the previous frame detection result. If the IUGR is greater than or equal to a preset third threshold, determine that the trajectory in the current detection result and the trajectory in the previous frame detection result corresponding to the IUGR belong to the same target human body.

[0065] For each low-confidence trajectory in the current frame detection result whose confidence is less than the confidence threshold, if the intersection-over-union ratio (IoU) corresponding to the low-confidence trajectory is less than the third threshold, the current frame is determined to be a mismatched frame of the low-confidence trajectory, and the mismatched frame count of the low-confidence trajectory is incremented by 1.

[0066] If the number of mismatched frames of the low-confidence trajectory is greater than or equal to a preset fourth threshold, the low-confidence trajectory will be removed.

[0067] Optionally, the spatial clipping module is specifically used to: calculate the height of the retained area. ; and calculate the width of the reserved area. Where H is the height of the bounding box, W is the width of the bounding box, upperbody_ratio is the ratio of the upper body of the human body, and environment_ratio is the ratio of the environment expansion.

[0068] By cropping and removing the regions outside the reserved areas corresponding to each target human body in each frame, a spatiotemporally segmented short temporal video is obtained.

[0069] Optionally, the video human behavior recognition system further includes:

[0070] The behavior assessment module is used to calculate the behavior assessment indicators of the target human body and determine the behavior evaluation level of the target human body based on the action sequence of the target human body.

[0071] Optionally, the video is an instructional video, the target human body is a student, the behavioral assessment indicator of the target human body is student behavioral engagement, the behavioral evaluation level of the target human body is learning engagement level, the action sequence of the target human body includes the sequence of various student behaviors, the various student behaviors include various behaviors that deviate from learning, and the behavior assessment module is specifically used for:

[0072] The total class time is obtained based on the duration of the video.

[0073] The duration and frequency of the student's disengagement behavior are determined based on the action sequence of the target human body.

[0074] Calculate students' original learning behavior input ,in, Indicates the total class time. Indicates the first The duration of the detachment from learning behavior, Indicates the first The number of times a disengagement behavior occurs, where M represents the total number of categories of disengagement behaviors;

[0075] Calculate the student's behavioral engagement ,in, The average initial learning engagement of all students. , Let N be the initial learning engagement level of the r-th student, and N be the number of students. The standard deviation of the distribution of all students' initial learning engagement is represented by the following: ;

[0076] If the student's behavioral engagement is greater than or equal to a preset fifth threshold, the learning engagement level is determined to be a high engagement level; if the student's behavioral engagement is less than the fifth threshold, the learning engagement level is determined to be a low engagement level.

[0077] In a third aspect of this application, an electronic device is provided, comprising:

[0078] At least one processor; and

[0079] A memory communicatively connected to at least one of the processors; wherein,

[0080] The memory stores instructions that can be executed by the processor to implement the video human behavior recognition method described above.

[0081] In a fourth aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for execution by the computer to implement the above-described video human behavior recognition method.

[0082] In a fifth aspect of this application, a computer program product comprising instructions is provided, which, when executed by a computer device, causes the computer device to perform the video human behavior recognition method described above.

[0083] The video human behavior recognition method and system provided in this application first process video frames through temporal segmentation and spatial cropping, and then integrates a token pruning and fusion token lightweighting strategy into the VideoMAE model. While ensuring recognition accuracy, it significantly reduces the computational load of Vision Transformer, improves adaptability to computing-constrained environments, enhances behavior recognition speed and deployment convenience, and can automatically identify fine-grained behaviors and complete indicator quantification and analysis, providing objective and reliable data support for scene behavior evaluation, such as student behavior analysis and evaluation in educational scenarios. Attached Figure Description

[0084] Figure 1 This is a flowchart illustrating one implementation of the video human behavior recognition method of this application;

[0085] Figure 2 This is a flowchart illustrating another implementation of the video human behavior recognition method of this application;

[0086] Figure 3 This is a flowchart illustrating one embodiment of step S205 in another implementation of the video human behavior recognition method of this application.

[0087] Figure 4This is a structural block diagram of one embodiment of the video human behavior recognition system of this application;

[0088] Figure 5 This is a structural block diagram of another embodiment of the video human behavior recognition system of this application. Detailed Implementation

[0089] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description, in conjunction with the accompanying drawings and examples, further clarifies this application. It should be understood that the specific examples described herein are merely illustrative and not intended to limit the scope of this application. Furthermore, the technical features involved in the various embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0090] The present application will now be described in detail with reference to the accompanying drawings. A first aspect of this application provides a video human behavior recognition method. Figure 1 The diagram illustrates a flowchart of one embodiment of the video human behavior recognition method of this application. Figure 1 As shown, the video human behavior recognition method of the first embodiment of this application includes:

[0091] Step S101: Perform human body tracking on the video to obtain the temporal trajectory of the target human body. The temporal trajectory includes the coordinates of the bounding box of the target human body, the video frame number, and the trajectory ID.

[0092] Step S102: Divide the video into segments according to a preset duration to obtain continuous short time-series videos and short time-series trajectories corresponding to the short time-series videos;

[0093] Step S103: Based on the width and height of the bounding box of each target human body in each frame of the short temporal video, according to the preset human upper body ratio and environment expansion ratio, the reserved area corresponding to each target human body in each frame is obtained, and the area outside the reserved area corresponding to each target human body in each frame is cropped to obtain the spatiotemporal segmented short temporal video.

[0094] Step S104: Using the VideoMAE model based on visual Transformer, identify the actions of the target human body in the spatiotemporally segmented short time-series video to obtain the action time sequence of the target human body. The VideoMAE model includes at least 4 Transformer layers.

[0095] Specifically, in step S101, the ByteTrack algorithm is used to perform human body tracking on the video. The ByteTrack algorithm is a mature algorithm, so its details will not be elaborated further. The ByteTrack algorithm can track multiple people simultaneously in complex scenes, such as teachers and students in a teaching setting. Even in cases of occlusion, overlap, or temporary loss of the target, it can maintain trajectory continuity and label consistency. ByteTrack consists of two stages: detection and association.

[0096] In one possible implementation, the YOLOX-X model based on an anchor-free architecture is used in the detection phase of the ByteTrack algorithm, with a confidence threshold of 0.5. In the association phase of the ByteTrack algorithm, the detection result of the current frame output from the detection phase is matched with the detection result of the previous frame. Specifically, matching the detection result of the current frame output from the detection phase with the detection result of the previous frame includes:

[0097] Calculate the IoU (Intersection over Union) between each trajectory in the current detection result and each trajectory in the previous frame detection result. Each trajectory in the current detection result corresponds to multiple IoU ratios. If the IoU ratio is greater than or equal to a preset third threshold (e.g., 0.8), it is determined that the trajectory in the current detection result corresponding to the IoU ratio and the trajectory in the previous frame detection result belong to the same target human body.

[0098] For each low-confidence trajectory in the current frame detection result whose confidence is less than the confidence threshold (e.g., 0.5), if the intersection-union ratio corresponding to the low-confidence trajectory is less than the third threshold (e.g., 0.8), the current frame is determined to be a mismatched frame of the low-confidence trajectory, and the number of mismatched frames of the low-confidence trajectory is incremented by 1. If the number of mismatched frames of the low-confidence trajectory is greater than or equal to a preset fourth threshold (e.g., 30 frames), the low-confidence trajectory is removed.

[0099] In other words, for high-confidence trajectories, such as those with a confidence level greater than or equal to 0.5, only one round of Intersection over Union (IoU) calculation is needed to determine whether they belong to the same target human body as some trajectories in the previous frame. If the intersection-over-union ratios corresponding to high-confidence trajectories are all less than the third threshold, it means that there are no trajectories in the previous frame that belong to the same target human body as the high-confidence trajectory, but the high-confidence trajectory is still retained. For low-confidence trajectories, such as those with a confidence level less than 0.5, if the cross-union ratio (CUN) of the corresponding trajectory is greater than or equal to the third threshold, it indicates that there was a trajectory with the same target human body as the low-confidence trajectory in the previous frame, so the low-confidence trajectory is retained. If the corresponding CUNs are all less than the third threshold, that is, there was no trajectory with the same target human body as the low-confidence trajectory in the previous frame, the current frame is determined as the mismatch frame of the low-confidence trajectory, and the number of mismatch frames of the low-confidence trajectory is incremented by 1. It is also determined whether the number of mismatch frames corresponding to the low-confidence trajectory is greater than or equal to the preset fourth threshold (e.g., 30), that is, whether the number of mismatch frames has exceeded the limit. If so, the low-confidence trajectory is removed from the trajectory pool, thereby reducing invalid trajectories and improving the target human body tracking efficiency. If the number of mismatch frames has not exceeded the limit, the low-confidence trajectory is retained.

[0100] Step S101 uses the ByteTrack algorithm to perform human body tracking on the video and outputs the temporal trajectory of the target human body. The temporal trajectory includes the coordinates of the bounding box of the target human body, the video frame number, and the trajectory ID.

[0101] Specifically, in step S102, the video is time-segmented. Specifically, the video is divided into segments according to a preset duration to obtain multiple consecutive, non-overlapping short time-series videos. Correspondingly, short time-series trajectories corresponding to these short time-series videos are also obtained. For example, if the preset duration is 4 seconds, then each short time-series video has a duration of 4 seconds. If the video frame rate is f (e.g., 25fps), then each short time-series video contains 4×f frames. If the total number of frames in the video is Z, then Z / (4f) short time-series videos can be obtained. In one possible implementation, empty frames without a target human body in the video can be removed before time-segmentation, or empty frames without a target human body in each short time-series video can be deleted after time-segmentation to reduce processing resource waste and improve the efficiency of the recognition process. By time-segmenting the video, it is easier to generate short time-series videos containing a single target subject and a single behavior, thereby improving the learning ability of the action recognition model for local behavioral features.

[0102] Specifically, in step S103, the short temporal video obtained by time segmentation is further spatially cropped. Specifically, based on the width and height of the bounding boxes of each target human body in each frame of the short temporal video, and according to the preset proportions of the upper body of the human body and the environment expansion ratio, the retention region corresponding to each target human body in each frame is obtained. Specifically, the center of the retention region coincides with the center of the bounding box, and the height of the retention region is calculated. ; and calculate the width of the reserved area. Where H is the height of the bounding box, W is the width of the bounding box, upperbody_ratio is the proportion of the upper body of the human body, and environment_ratio is the environment expansion ratio. After obtaining the retained region, the region outside the retained region corresponding to each target human body in each frame is cropped to obtain a spatiotemporally segmented short temporal video. In one possible implementation, if the cropped region exceeds the video boundary, the system can automatically perform boundary truncation. Spatial cropping can effectively reduce scene redundancy information, improve model recognition and inference efficiency, and also help enhance the action recognition model's ability to learn local behavioral features.

[0103] Specifically, in step S104, the VideoMAE model based on the Vision Transformer, namely VideoMAE ViT-B (Vision Transformer Base), is used as the action recognition model. The VideoMAE model includes at least 4 Transformer layers. The VideoMAE model is a mature model, so the model itself will not be described in detail. The number of Transformer layers in the VideoMAE model can be configured according to actual needs. In one possible implementation, the input size of the VideoMAE model is 16×224×224, that is, each input contains 16 video frame sequences, each frame resolution is 224×224, the Patch Embedding size is 16×16, and the model contains 12 Transformer layers, for example, composed of 12 stacked Transformer Blocks. Each Transformer layer includes a multi-head self-attention mechanism and a feedforward network structure.

[0104] To reduce computational load and improve the inference speed of the VideoMAE model, in one embodiment of step S104 of the video human behavior recognition method of this application, for each frame of the spatiotemporally segmented short temporal video, based on the token set of each frame... , collect tokens Tokens in odd positions are grouped into a token set TA1. The tokens in even positions of the list are grouped into a token set TA2. The feature similarity between the tokens in set TA1 and the tokens in set TA2 is calculated. Where S(i,j) is the feature similarity between the i-th token in set TA1 and the j-th token in set TA2, 1≤i≤n, 1≤j≤n;

[0105] For each token, determine its corresponding feature similarity. Check if it is greater than a preset first threshold (e.g., 0.9), and calculate its corresponding feature similarity. The number of feature similarities S(i,j) greater than the first threshold is the feature similarity number. If the feature similarity number is greater than the preset second threshold (e.g., 3), the token corresponding to the feature similarity number is added to the pre-cut token set R of the corresponding frame (i.e. the video frame where the corresponding token is located), and the tokens in the pre-cut token set R are sorted from largest to smallest according to the corresponding feature similarity number.

[0106] The final number of pruned tokens r for each frame is obtained by calculating the average number of tokens in the pre-pruned token set R for all frames. ,in, The total number of tokens in the set R of tokens to be pruned for all frames, and F is the total number of frames.

[0107] For each frame, delete the first r tokens from the pre-pruning token set R, and generate a pruned token set from the remaining tokens. Where m≤n, all remaining tokens include the remaining tokens in the set of tokens to be pruned R and the tokens in the frame that are not in the set of tokens to be pruned. To prevent over-pruning, an upper limit on the number of tokens to be deleted can be set (e.g., 64). If r is greater than the upper limit, then the upper limit number will be deleted.

[0108] The token set of each frame of the spatiotemporally segmented short-sequence video. Input the first Transformer layer of the VideoMAE model to obtain the set of pruned tokens for each frame of the spatiotemporally segmented short temporal video. The input to the second to Cth Transformer layers of the VideoMAE model is further used for feature encoding, ultimately identifying the actions of the target human body in the spatiotemporally segmented short-sequence video, thus obtaining the temporal sequence of the target human body's actions, where C is the number of the Transformer layers in the VideoMAE model. In practical applications, after pruning, the number of input tokens per frame can be reduced by approximately 20%–30%, significantly reducing the computational overhead of the Transformer layers. The token pruning stage can remove low-level visual repetition areas, such as the blackboard background and fixed desktop in a teaching scene, thereby concentrating computational resources on the human body and interactive action areas.

[0109] In another embodiment of step S104 of the video human behavior recognition method of this application, the tokens from the 4th layer to the last Transformer layer of the input VideoMAE model are fused to compress the token representation and maintain the semantic integrity of higher layers (after the 4th layer). Specifically, the token set of each frame of the h-th Transformer layer of the input VideoMAE model is fused. , collect tokens Tokens in odd positions in the sequence are grouped into a token set. , collect tokens Tokens in even positions in the sequence are grouped into a token set. Calculate the set tokens and sets Feature similarity between tokens Where S(i,j) is a set The i-th token and the set The feature similarity between the j-th tokens is 1≤i≤k, 1≤j≤k; where 4≤h≤C, and C is the number of Transformer layers (e.g., 12); if a pruning step is performed in step S104, then k≤m; if no pruning step is performed, then k≤n.

[0110] For each frame, sort all tokens in descending order according to their corresponding feature similarity S(i,j), and generate similar token pairs (e.g., S(i,j) corresponding to...). and Given a sequence of token pairs, the first r (equal to the final number of tokens removed) token pairs in the sequence of similar token pairs, i.e., the highly similar token pairs, are weighted and averaged according to their token size to generate a merged token. ,in and They are tokens and token Size, token and token Belongs to the highly similar token pair, For the fusion token, The size is ;

[0111] For the fusion token, the Transformer layer attention A Where Q is the key vector of the fused token, K is the value vector of the fused token, d is the vector dimension of the fused token, and s is the size of the fused token; for non-fused tokens, the attention calculation is performed according to the conventional Transformer layer attention calculation method;

[0112] Replace the token set with the fused token. The corresponding token pairs in the set yield a fusion token set. ;

[0113] The fusion token set Input the Multi-Layer Perceptron (MLP) submodule of the h-th Transformer layer. It should be noted that the input is the token set of the h-th Transformer layer. Input the other sub-modules of the h-th Transformer layer and merge the token set. The input is the MLP submodule of the h-th Transformer layer, which further encodes features to ultimately identify the actions of the target human body in the spatiotemporally segmented short temporal video, obtaining the temporal sequence of the target human body's actions. Token fusion is restricted to within the same time frame to maintain temporal consistency and prevent action temporal disorder caused by cross-frame fusion. Token fusion is applied to each layer from the fourth Transformer layer to the last Transformer layer, thereby achieving high semantic-level feature compression and reducing the number of tokens while preserving discriminative features. It is evident that the token fusion stage can also reduce computational cost and improve the inference speed of the VideoMAE model. In practical applications, the token fusion stage can reduce the number of tokens by approximately 30% and reduce the overall computational cost by more than 50%.

[0114] In one possible implementation of step S104 of the video human behavior recognition method of this application, step S104 only includes a token pruning stage and does not include a token fusion stage. In another possible implementation of step S104 of the video human behavior recognition method of this application, step S104 only includes a token fusion stage and does not include a token pruning stage. In yet another possible implementation of step S104 of the video human behavior recognition method of this application, step S104 includes both a token pruning stage and a token fusion stage, thereby constituting a lightweight two-stage token strategy for the VideoMAE model, which can significantly reduce computational load and improve model recognition efficiency while preserving mutability features.

[0115] It should be noted that when using the VideoMAE model for action recognition, the action categories to be recognized can be set according to the specific application scenario. The VideoMAE model's final classification head (MLP Head) outputs the action time sequence of various actions of the target human body. Taking the teaching scenario as an example, the target human body is the student, and various actions include raising hands, discussing, practicing, standing up to answer, etc. The VideoMAE model outputs the action time sequence of raising hands, the action time sequence of discussing, etc.

[0116] Figure 2 A flowchart illustrating another embodiment of the video human behavior recognition method of this application is shown, such as... Figure 2 As shown, the video human behavior recognition method of the second embodiment of this application includes:

[0117] Step S201: Perform human body tracking on the video to obtain the temporal trajectory of the target human body. The temporal trajectory includes the coordinates of the bounding box of the target human body, the video frame number, and the trajectory ID.

[0118] Step S202: Divide the video into segments according to a preset duration to obtain continuous short time-series videos and short time-series trajectories corresponding to the short time-series videos;

[0119] Step S203: Based on the width and height of the bounding box of each target human body in each frame of the short temporal video, according to the preset human upper body ratio and environment expansion ratio, the reserved area corresponding to each target human body in each frame is obtained, and the area outside the reserved area corresponding to each target human body in each frame is cropped to obtain the spatiotemporal segmented short temporal video.

[0120] Step S204: Using the VideoMAE model based on visual Transformer, identify the actions of the target human body in the spatiotemporally segmented short temporal video to obtain the action temporal sequence of the target human body. The VideoMAE model includes at least 4 Transformer layers.

[0121] Step S205: Calculate the behavioral assessment index of the target human body and determine the behavioral evaluation level of the target human body based on the action sequence of the target human body.

[0122] Specifically, steps S201-S204 can refer to steps S101-S104.

[0123] Specifically, in step S205, after the action sequence of the target human body is identified using the VideoMAE model, the target human body's behavior can be further analyzed based on the action sequence of the target human body according to the actual application scenario, and the behavioral evaluation index of the target human body can be calculated and the behavioral evaluation level of the target human body can be determined.

[0124] For example, the application scenario is a teaching scenario, the video is a teaching video, the target human body is a student, the behavioral assessment index of the target human body is the student's behavioral engagement, the behavioral evaluation level of the target human body is the learning engagement level, and the action sequence of the target human body includes the sequence of various student behaviors (such as raising hands, discussing, practicing, standing up to answer, drowsiness, small movements, etc.), and the various student behaviors include various behaviors that deviate from learning (such as drowsiness, small movements, etc.). Figure 3 This diagram illustrates a flowchart of one embodiment of step S205 in another implementation of the video human behavior recognition method of this application. Specifically, as shown... Figure 3 As shown, step S205 may include:

[0125] Step S2051: Obtain the total class duration based on the video length;

[0126] Step S2052: Determine the duration and frequency of the student's disengagement behavior based on the action sequence of the target human body in the video;

[0127] Step S2053: Calculate the students' original learning behavior engagement. ,in, Indicates the total class time. Indicates the first The duration of the detachment from learning behavior, Indicates the first The number of times a disengagement behavior occurs, where M represents the total number of categories of disengagement behaviors;

[0128] Step S2054: Calculate student behavioral engagement. ,in, The average initial learning engagement of all students. , Let N be the initial learning engagement level of the r-th student, and N be the number of students. The standard deviation of the distribution of all students' initial learning engagement is represented by the following: ;

[0129] Step S2055: If the student's behavioral engagement is greater than or equal to a preset fifth threshold, the learning engagement level is determined to be a high engagement level.

[0130] Step S2056: If the student's behavioral engagement is less than the fifth threshold, the learning engagement level is determined to be a low engagement level.

[0131] Specifically, the total class time can be obtained from the length of the classroom video; the time sequence of students' actions can be used to determine the time sequence of disengagement behaviors, such as drowsiness and fidgeting. The duration and frequency of these disengagement behaviors can then be determined from this time sequence. By combining the total class time, the duration and frequency of student disengagement behaviors, and the pre-defined total number of disengagement behavior categories based on the assessment needs, the students' initial engagement level can be calculated. ,in, Indicates the total class time. Indicates the first The duration of the detachment from learning behavior, Indicates the first The number of times a student engages in a learning activity, M, represents the total number of categories of such behaviors. A higher initial engagement score indicates more focused and engaged student behavior; a lower score indicates more distracted or passive behavior. Student engagement can be calculated from the initial engagement score. ,in, The average initial learning engagement of all students. , Let N be the initial learning engagement level of the r-th student, and N be the number of students. The standard deviation of the distribution of all students' initial learning engagement is represented by the following: .

[0132] Specifically, after calculating the student's behavioral engagement level, the student's learning engagement level can be determined based on this level, thus understanding the student's level of engagement in learning behavior. Specifically, if the student's behavioral engagement level is greater than or equal to a preset fifth threshold (e.g., -0.84), the learning engagement level is determined to be high; if the student's behavioral engagement level is less than the fifth threshold, the learning engagement level is determined to be low. In one possible implementation, the student's behavioral engagement level can be further calculated continuously over time, and an engagement curve can be plotted to reflect the dynamic changes in student concentration during the classroom process. In another possible implementation, the calculated results and engagement curve can be visualized, facilitating a direct observation of students' learning progress in the classroom. Therefore, by automatically identifying the fine-grained action characteristics of students and teachers, objective and continuous quantitative analysis of large-scale teaching scenarios can be achieved, providing objective data support for educational indicators such as classroom engagement, teacher-student interaction quality, and learning participation levels.

[0133] The video human behavior recognition method provided in this application, through spatiotemporal segmentation and cropping of video frames, and the integration of token pruning and / or fusion with a token lightweighting strategy in the VideoMAE model, can significantly reduce the computational load of the Vision Transformer while ensuring recognition accuracy. This improves adaptability to computationally limited environments and enhances behavior recognition efficiency. Furthermore, the method can be implemented modularly step-by-step, further improving deployment convenience. Therefore, it can achieve automated recognition of fine-grained human behavior in application scenarios and complete indicator quantification and analysis, providing objective and reliable data support for scenario behavior evaluation, such as student behavior analysis and evaluation in educational settings.

[0134] A second aspect of this application provides a video human behavior recognition system. Figure 4 A structural block diagram of one embodiment of the video human behavior recognition system of this application is shown, as follows: Figure 4 As shown, the video human behavior recognition system of the third embodiment of this application includes:

[0135] The human body tracking module 401 is used to perform human body tracking on the video to obtain the temporal trajectory of the target human body. The temporal trajectory includes the coordinates of the bounding box of the target human body, the video frame number, and the trajectory ID.

[0136] The time segmentation module 402 is used to segment the video according to a preset duration to obtain continuous short time-series videos and short time-series trajectories corresponding to the short time-series videos;

[0137] The spatial cropping module 403 is used to obtain the reserved area corresponding to each target human body in each frame according to the width and height of the bounding box of each target human body in each frame of the short temporal video, according to the preset human upper body ratio and environment expansion ratio, and to crop and remove the area outside the reserved area corresponding to each target human body in each frame to obtain the spatiotemporally cut short temporal video.

[0138] The action recognition module 404 is used to recognize the action of the target human body in the spatiotemporally segmented short time-series video using the VideoMAE model based on visual Transformer, and to obtain the action time sequence of the target human body. The VideoMAE model includes at least 4 Transformer layers.

[0139] Specifically, the human body tracking module is used for:

[0140] The ByteTrack algorithm is used to perform human body tracking on the video.

[0141] In the detection phase of the ByteTrack algorithm, the YOLOX-X model based on the Anchor-free architecture is used, with a confidence threshold of 0.5.

[0142] In the association phase of the ByteTrack algorithm, the detection result of the current frame in the detection phase is matched with the detection result of the previous frame;

[0143] When the human body tracking module matches the current frame detection result with the previous frame detection result during the detection phase, it is specifically used for:

[0144] Calculate the intersection-union ratio (IUGR) of each trajectory in the current detection result with each trajectory in the previous frame detection result. If the IUGR is greater than or equal to a preset third threshold, determine that the trajectory in the current detection result and the trajectory in the previous frame detection result corresponding to the IUGR belong to the same target human body.

[0145] For each low-confidence trajectory in the current frame detection result whose confidence is less than the confidence threshold, if the intersection-over-union ratio (IoU) corresponding to the low-confidence trajectory is less than the third threshold, the current frame is determined to be a mismatched frame of the low-confidence trajectory, and the mismatched frame count of the low-confidence trajectory is incremented by 1.

[0146] If the number of mismatched frames of the low-confidence trajectory is greater than or equal to a preset fourth threshold, the low-confidence trajectory will be removed.

[0147] Specifically, the spatial clipping module is used to: calculate the height of the retained area. ; and calculate the width of the reserved area. Where H is the height of the bounding box, W is the width of the bounding box, upperbody_ratio is the ratio of the upper body of the human body, and environment_ratio is the ratio of the environment expansion.

[0148] By cropping and removing the regions outside the reserved areas corresponding to each target human body in each frame, a spatiotemporally segmented short temporal video is obtained.

[0149] Specifically, in one possible implementation, the action recognition module is used to:

[0150] For each frame of the spatiotemporally segmented short temporal video, based on the token set of each frame... , collect tokens Tokens in odd positions are grouped into a token set TA1. The tokens in even positions of the list are grouped into a token set TA2. The feature similarity between the tokens in set TA1 and the tokens in set TA2 is calculated. Where S(i,j) is the feature similarity between the i-th token in set TA1 and the j-th token in set TA2, 1≤i≤n, 1≤j≤n;

[0151] For each token, calculate the number of feature similarities S(i,j) greater than a preset first threshold in the feature similarity S(i,j) corresponding to the token, i.e. the feature similarity count. If the feature similarity count is greater than a preset second threshold, add the token corresponding to the feature similarity count to the pre-pruning token set R of the corresponding frame, and sort the tokens in the pre-pruning token set R in descending order of the corresponding feature similarity count.

[0152] The final number of tokens to be pruned in each frame is calculated by averaging the number of tokens in the pre-pruned token set R for all frames.

[0153] For each frame, remove the first r tokens from the pre-pruning token set R, and generate a pruned token set from the remaining tokens. Where m ≤ n;

[0154] The token set of each frame of the spatiotemporally segmented short-sequence video. Input the first Transformer layer of the VideoMAE model to obtain the set of pruned tokens for each frame of the spatiotemporally segmented short temporal video. Input the second to the Cth Transformer layers of the VideoMAE model to identify the actions of the target human body in the spatiotemporally segmented short time-series video, and obtain the action time sequence of the target human body, where C is the number of the Transformer layers of the VideoMAE model.

[0155] Specifically, in one embodiment, the action recognition module is further configured to:

[0156] The set of tokens for each frame of the h-th Transformer layer input to the VideoMAE model. , collect tokens Tokens in odd positions in the sequence are grouped into a token set. , collect tokens Tokens in even positions in the sequence are grouped into a token set. Calculate the set tokens and sets Feature similarity between tokens Where S(i,j) is a set The i-th token and the set The feature similarity between the j-th tokens is 1≤i≤k, 1≤j≤k; where k≤m, 4≤h≤C, and C is the number of Transformer layers.

[0157] For each frame, all tokens are sorted in descending order of their corresponding feature similarity S(i,j) to generate a sequence of similar token pairs. The first r highly similar token pairs in the sequence are then weighted and averaged according to their token size to generate a fused token. ,in and They are tokens and token Size, token and token Belongs to the highly similar token pair, For the fusion token, The size is ;

[0158] For the fusion token, the Transformer layer attention A Where Q is the key vector of the fusion token, K is the value vector of the fusion token, d is the vector dimension of the fusion token, and s is the size of the fusion token;

[0159] Replace the token set with the fused token. The corresponding token pairs in the set yield a fusion token set. ;

[0160] The fusion token set Input the multilayer perceptron submodule of the h-th Transformer layer to identify the actions of the target human body in the spatiotemporally segmented short time-series video and obtain the action time sequence of the target human body.

[0161] It should be noted that the action recognition module can perform only token pruning, only token fusion, or both during operation.

[0162] Figure 5 A structural block diagram of another embodiment of the video human behavior recognition system of this application is shown, such as... Figure 5 As shown, the video human behavior recognition system of the third embodiment of this application includes:

[0163] The human body tracking module 501 is used to perform human body tracking on the video to obtain the temporal trajectory of the target human body. The temporal trajectory includes the coordinates of the bounding box of the target human body, the video frame number, and the trajectory ID.

[0164] The time segmentation module 502 is used to segment the video according to a preset duration to obtain continuous short time-series videos and short time-series trajectories corresponding to the short time-series videos;

[0165] The spatial cropping module 503 is used to obtain the reserved area corresponding to each target human body in each frame according to the width and height of the bounding box of each target human body in each frame of the short temporal video, according to the preset human upper body ratio and environment expansion ratio, and to crop and remove the area outside the reserved area corresponding to each target human body in each frame to obtain the spatiotemporally cut short temporal video.

[0166] The action recognition module 504 is used to recognize the action of the target human body in the spatiotemporally segmented short temporal video using a VideoMAE model based on visual Transformer, and to obtain the action temporal sequence of the target human body. The VideoMAE model includes at least 4 Transformer layers.

[0167] The behavior assessment module 505 is used to calculate the behavior assessment index of the target human body and determine the behavior evaluation level of the target human body based on the action sequence of the target human body.

[0168] For example, the video is an instructional video, the target human body is a student, the behavioral assessment indicator of the target human body is student behavioral engagement, the behavioral evaluation level of the target human body is learning engagement level, the action sequence of the target human body includes the sequence of various student behaviors, the various student behaviors include various behaviors that deviate from learning, and the behavior assessment module is specifically used for:

[0169] The total class time is obtained based on the duration of the video.

[0170] The duration and frequency of the student's disengagement behavior are determined based on the action sequence of the target human body.

[0171] Calculate students' original learning behavior input ,in, Indicates the total class time. Indicates the first The duration of the detachment from learning behavior, Indicates the first The number of times a disengagement behavior occurs, where M represents the total number of categories of disengagement behaviors;

[0172] Calculate the student's behavioral engagement ,in, The average initial learning engagement of all students. , Let N be the initial learning engagement level of the r-th student, and N be the number of students. The standard deviation of the distribution of all students' initial learning engagement is represented by the following: ;

[0173] If the student's behavioral engagement is greater than or equal to a preset fifth threshold, the learning engagement level is determined to be a high engagement level; if the student's behavioral engagement is less than the fifth threshold, the learning engagement level is determined to be a low engagement level.

[0174] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the system described above can be referred to the corresponding processes in the foregoing method embodiments, and therefore will not be repeated here.

[0175] The video human behavior recognition system provided in this application combines video frame spatiotemporal segmentation and cropping with the token lightweighting strategy of the VideoMAE model. While ensuring recognition accuracy, it significantly reduces computational load, adapts to environments with limited computing power, and improves behavior recognition efficiency. The system is modular and can be implemented in stages, making deployment more convenient. It can be effectively applied to real-world scenarios, automatically identifying and quantifying fine-grained human behaviors, providing objective data support for behavior assessment in educational and other scenarios.

[0176] It should be noted that the video human behavior recognition system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of this application can be further decomposed or combined. For example, the modules in the above embodiments can be merged into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of this application are only for distinguishing the various modules or steps and are not considered as an improper limitation of this application.

[0177] In a third aspect of this application, an electronic device is also provided, the electronic device comprising: at least one processor; and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to implement the above-described video human behavior recognition method.

[0178] In a fourth aspect of this application, a computer-readable storage medium is also provided, the computer-readable storage medium storing computer instructions for execution by the computer to implement the above-described video human behavior recognition method.

[0179] In a fifth aspect of this application, a computer program product comprising instructions is also provided, which, when executed by a computer device, causes the computer device to perform the video human behavior recognition method described above.

[0180] Those skilled in the art will recognize that the modules and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. The programs corresponding to the software modules and method steps can be placed in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. To clearly illustrate the interchangeability of electronic hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0181] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0182] The terms “first”, “second”, etc., are used to distinguish similar objects, not to describe or indicate a specific order or sequence.

[0183] The term "comprising" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent in such process, method, article, or apparatus / device.

[0184] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A video human behavior recognition method, characterized in that, include: Human body tracking is performed on the video to obtain the temporal trajectory of the target human body. The temporal trajectory includes the coordinates of the bounding box of the target human body, the video frame number, and the trajectory ID. The video is segmented according to a preset duration to obtain continuous short time-series videos and short time-series trajectories corresponding to the short time-series videos; Based on the width and height of the bounding box of each target human body in each frame of the short temporal video, and according to the preset human upper body ratio and environment expansion ratio, the reserved area corresponding to each target human body in each frame is obtained, and the area outside the reserved area corresponding to each target human body in each frame is cropped to obtain the spatiotemporal segmented short temporal video. The VideoMAE model, based on a visual Transformer, is used to identify the actions of the target human body in the spatiotemporally segmented short time-series video, thereby obtaining the action time sequence of the target human body. The VideoMAE model includes at least four Transformer layers.

2. The method as described in claim 1, characterized in that, The method uses the VideoMAE model based on the visual Transformer to identify the actions of the target human body in the spatiotemporally segmented short temporal video, obtaining the temporal sequence of the target human body's actions, including: For each frame of the spatiotemporally segmented short temporal video, based on the token set of each frame... , collect tokens Tokens in odd positions are grouped into a token set TA1. The tokens in even positions of the list are grouped into a token set TA2. The feature similarity between the tokens in set TA1 and the tokens in set TA2 is calculated. Where S(i,j) is the feature similarity between the i-th token in set TA1 and the j-th token in set TA2, 1≤i≤n, 1≤j≤n; For each token, calculate the feature similarity count of the token. The feature similarity count is the number of feature similarities S(i,j) of the token that are greater than a preset first threshold. If the feature similarity count is greater than a preset second threshold, add the token corresponding to the feature similarity count to the pre-pruning token set R of the corresponding frame, and sort the tokens in the pre-pruning token set R in descending order of the corresponding feature similarity count. The final number of tokens to be pruned in each frame is calculated by averaging the number of tokens in the pre-pruned token set R for all frames. For each frame, remove the first r tokens from the pre-pruning token set R, and generate a pruned token set from the remaining tokens. Where m ≤ n; The token set of each frame of the spatiotemporally segmented short-sequence video. Input the first Transformer layer of the VideoMAE model to obtain the set of pruned tokens for each frame of the spatiotemporally segmented short temporal video. Input the second to the Cth Transformer layers of the VideoMAE model to identify the actions of the target human body in the spatiotemporally segmented short time-series video, and obtain the action time sequence of the target human body, where C is the number of the Transformer layers of the VideoMAE model.

3. The method as described in claim 1 or 2, characterized in that, The method of using the VideoMAE model based on the visual Transformer to identify the movements of the target human body in the spatiotemporally segmented short-sequence video also includes: The set of tokens for each frame of the h-th Transformer layer input to the VideoMAE model. , collect tokens Tokens in odd positions in the sequence are grouped into a token set. , collect tokens Tokens in even positions in the sequence are grouped into a token set. Calculate the set tokens and sets Feature similarity between tokens Where S(i,j) is a set The i-th token and the set The feature similarity between the j-th tokens is 1≤i≤k, 1≤j≤k; where k≤m, 4≤h≤C, and C is the number of Transformer layers. For each frame, all tokens are sorted in descending order of their corresponding feature similarity S(i,j) to generate a sequence of similar token pairs. The first r highly similar token pairs in the sequence are then weighted and averaged according to their token size to generate a fused token. ,in and They are tokens and token Size, token and token Belongs to the highly similar token pair, For the fusion token, The size is ; For the fusion token, the Transformer layer attention A Where Q is the key vector of the fusion token, K is the value vector of the fusion token, d is the vector dimension of the fusion token, and s is the size of the fusion token; Replace the token set with the fused token. The corresponding token pairs in the set yield a fusion token set. ; The fusion token set Input the multilayer perceptron submodule of the h-th Transformer layer to identify the actions of the target human body in the spatiotemporally segmented short time-series video and obtain the action time sequence of the target human body.

4. The method as described in claim 1, characterized in that, The process of tracking a human body in a video to obtain the temporal trajectory of the target human body includes: The ByteTrack algorithm is used to perform human body tracking on the video. In the detection phase of the ByteTrack algorithm, the YOLOX-X model based on the Anchor-free architecture is used, with a confidence threshold of 0.

5. In the association phase of the ByteTrack algorithm, the detection result of the current frame in the detection phase is matched with the detection result of the previous frame, including: Calculate the intersection-union ratio (IoU) between each trajectory in the current detection result and each trajectory in the previous frame detection result. If the IoU is greater than or equal to a preset third threshold, determine that the trajectory in the current detection result corresponding to the IoU and the trajectory in the previous frame detection result belong to the same target human body. For each low-confidence trajectory in the current frame detection result with a confidence level less than the confidence threshold, if the IoU corresponding to the low-confidence trajectory is less than the third threshold, determine that the current frame is a mismatch frame of the low-confidence trajectory, and increment the mismatch frame count of the low-confidence trajectory by 1. If the number of mismatched frames of the low-confidence trajectory is greater than or equal to a preset fourth threshold, the low-confidence trajectory will be removed.

5. The method as described in claim 1, characterized in that, The step of obtaining the reserved region corresponding to each target human body in each frame of the short-time video based on the width and height of the bounding box of each target human body in each frame, according to a preset human upper body ratio and environment expansion ratio, includes: calculating the height of the reserved region. ; and calculate the width of the reserved area. Where H is the height of the bounding box, W is the width of the bounding box, upperbody_ratio is the proportion of the upper body, and environment_ratio is the environment expansion ratio.

6. The method as described in claim 1, characterized in that, Also includes: Based on the action sequence of the target human body, behavioral assessment indicators and behavioral evaluation levels are calculated for the target human body. The video is an instructional video, the target human body is a student, the behavioral assessment indicator is student engagement, and the behavioral evaluation level is learning engagement level. The action sequence of the target human body includes the sequence of various student behaviors, including various behaviors that deviate from learning. The process of calculating the behavioral assessment indicators and determining the behavioral evaluation level based on the action sequence of the target human body includes: The total class time is obtained based on the video length; the duration and frequency of the student's disengagement behavior are determined based on the action sequence of the target human body. Calculate students' original learning behavior input ,in, Indicates the total class time. Indicates the first The duration of the detachment from learning behavior, Indicates the first The number of times a disengagement behavior occurs, where M represents the total number of categories of disengagement behaviors; Calculate the student's behavioral engagement ,in, The average initial learning engagement of all students. , Let N be the initial learning engagement level of the r-th student, and N be the number of students. The standard deviation of the distribution of all students' initial learning engagement is represented by the following: ; If the student's behavioral engagement is greater than or equal to a preset fifth threshold, the learning engagement level is determined to be a high engagement level; if the student's behavioral engagement is less than the fifth threshold, the learning engagement level is determined to be a low engagement level.

7. A video human behavior recognition system, characterized in that, include: The human body tracking module is used to track human bodies in videos and obtain the temporal trajectory of the target human body. The temporal trajectory includes the coordinates of the bounding box of the target human body, the video frame number, and the trajectory ID. The time segmentation module is used to divide the video into segments according to a preset duration to obtain continuous short time-series videos and short time-series trajectories corresponding to the short time-series videos; The spatial cropping module is used to obtain the reserved area corresponding to each target human body in each frame according to the width and height of the bounding box of each target human body in each frame of the short temporal video, according to the preset human upper body ratio and environment expansion ratio, and to crop and remove the area outside the reserved area corresponding to each target human body in each frame, so as to obtain the spatiotemporally cut short temporal video. The action recognition module is used to identify the actions of the target human body in the spatiotemporally segmented short time-series video using the VideoMAE model based on visual Transformer, and to obtain the action time sequence of the target human body. The VideoMAE model includes at least 4 Transformer layers.

8. The system as described in claim 7, characterized in that, The action recognition module is specifically used for: For each frame of the spatiotemporally segmented short temporal video, based on the token set of each frame... , collect tokens Tokens in odd positions are grouped into a token set TA1. The tokens in even positions of the list are grouped into a token set TA2. The feature similarity between the tokens in set TA1 and the tokens in set TA2 is calculated. Where S(i,j) is the feature similarity between the i-th token in set TA1 and the j-th token in set TA2, 1≤i≤n, 1≤j≤n; For each token, calculate the feature similarity count of the token. The feature similarity count is the number of feature similarities S(i,j) of the token that are greater than a preset first threshold. If the feature similarity count is greater than a preset second threshold, add the token corresponding to the feature similarity count to the pre-pruning token set R of the corresponding frame, and sort the tokens in the pre-pruning token set R in descending order of the corresponding feature similarity count. The final number of tokens to be pruned in each frame is calculated by averaging the number of tokens in the pre-pruned token set R for all frames. For each frame, remove the first r tokens from the pre-pruning token set R, and generate a pruned token set from the remaining tokens. Where m ≤ n; The token set of each frame of the spatiotemporally segmented short-sequence video. Input the first Transformer layer of the VideoMAE model to obtain the set of pruned tokens for each frame of the spatiotemporally segmented short temporal video. Input the second to the Cth Transformer layers of the VideoMAE model to identify the actions of the target human body in the spatiotemporally segmented short temporal video and obtain the temporal sequence of the target human body's actions, where C is the number of the Transformer layers of the VideoMAE model; The action recognition module is also used for: The set of tokens for each frame of the h-th Transformer layer input to the VideoMAE model. , collect tokens Tokens in odd positions in the sequence are grouped into a token set. , collect tokens Tokens in even positions in the sequence are grouped into a token set. Calculate the set tokens and sets Feature similarity between tokens Where S(i,j) is a set The i-th token and the set The feature similarity between the j-th tokens is 1≤i≤k, 1≤j≤k; where k≤m, 4≤h≤C, and C is the number of Transformer layers. For each frame, all tokens are sorted in descending order of their corresponding feature similarity S(i,j) to generate a sequence of similar token pairs. The first r highly similar token pairs in the sequence are then weighted and averaged according to their token size to generate a fused token. ,in and They are tokens and token Size, token and token Belongs to the highly similar token pair, For the fusion token, The size is ; For the fusion token, the Transformer layer attention A Where Q is the key vector of the fusion token, K is the value vector of the fusion token, d is the vector dimension of the fusion token, and s is the size of the fusion token; Replace the token set with the fused token. The corresponding token pairs in the set yield a fusion token set. ; The fusion token set Input the multilayer perceptron submodule of the h-th Transformer layer to identify the actions of the target human body in the spatiotemporally segmented short time-series video and obtain the action time sequence of the target human body.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor to implement the video human behavior recognition method as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are executed by the computer to implement the video human behavior recognition method as described in any one of claims 1-6.

Images