Teacher atomic action recognition method and device based on improved end-to-end network

Abstract

This application belongs to the field of computer vision technology, specifically disclosing a method and apparatus for recognizing teacher atomic actions based on an improved end-to-end network. Through this application, based on an improved target end-to-end network, temporal and spatial video features are determined according to the current teaching video of the target teacher. After generating multi-window feature groups, a frame selection network is used to fuse the multi-window feature groups, and the atomic actions of the target teacher are recognized based on the fusion result. By embedding spatial adaptation units and temporal adaptation units into the end-to-end network, feature extraction is performed separately. During network training, memory consumption is reduced by training only the parameters of these units. Then, through multi-branch feature processing and adaptive fusion, the ability to discriminate complex temporal information in video action recognition is improved, thereby effectively improving the accuracy and efficiency of atomic action recognition, reducing memory requirements, and lowering deployment difficulty.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and more particularly, relates to a teacher atomic action recognition method and device based on an improved end-to-end network. BACKGROUND

[0002] Atomic action recognition is an important research direction in the fields of computer vision and artificial intelligence, and has been widely applied in various fields, such as education. At present, the common way of atomic action recognition is: (1) through a single-stage action recognition network to realize in an end-to-end manner, without splitting steps in the implementation process, spatial and temporal features are directly extracted from multiple video sequences through 3D convolution layers or "2D convolution + temporal attention" units of the model, but directly inputting the entire teaching video will be disturbed by complex backgrounds, resulting in misrecognition, which leads to low accuracy of atomic action recognition; (2) relying on a conventional end-to-end network, taking a Vision Transformer (ViT) as a backbone, and training through a mask masking manner, but the above network needs full-parameter fine-tuning when training, and has a high requirement for the display memory, which affects the device performance and leads to low efficiency of atomic action recognition. Therefore, the above method has low accuracy and efficiency in teacher atomic action recognition. SUMMARY

[0003] In view of the defects of the prior art, the purpose of the present application is to provide a teacher atomic action recognition method and device based on an improved end-to-end network, aiming to solve the problem of low accuracy and efficiency of teacher atomic action recognition caused by the interference of complex backgrounds, full-parameter fine-tuning during network training, and high requirement for display memory in the prior art.

[0004] To achieve the above purpose, in a first aspect, the present application provides a teacher atomic action recognition method based on an improved end-to-end network, comprising:

[0005] determining time-dimension video features and space-dimension video features according to a current teaching video of a target teacher based on an improved target end-to-end network, wherein the improved target end-to-end network comprises a space adaptation unit, a time sequence adaptation unit, and a parameter freezing unit;

[0006] generating a multi-window feature group according to the time-dimension video features and the space-dimension video features through a multi-time window attention mechanism;

[0007] fusing the multi-window feature group through a frame selection network, and recognizing atomic actions of the target teacher according to a fusion result.

[0008] In an embodiment, the step of determining the time-dimension video feature and the space-dimension video feature according to the current teaching video of the target teacher based on the improved target end-to-end network comprises:

[0009] Determining a current image block according to the current teaching video of the target teacher based on the previous conventional unit in the improved target end-to-end network;

[0010] Reshaping the current image block to obtain a current three-dimensional spatial structure image;

[0011] Upsampling the current three-dimensional spatial structure image based on the space adaptation unit in the improved target end-to-end network, and performing local feature extraction on the upsampled three-dimensional spatial structure image to obtain a local limb feature;

[0012] Downsampling the local limb feature to obtain a space-dimension video feature;

[0013] Upsampling the space-dimension video feature based on the time sequence adaptation unit in the improved target end-to-end network, and performing time sequence convolution on the upsampled continuous multi-frame three-dimensional feature to obtain a time sequence convolution feature;

[0014] Downsampling the time sequence convolution feature, and performing linear transformation on the downsampled time sequence feature to obtain a time-dimension video feature.

[0015] In an embodiment, before the step of determining the time-dimension video feature and the space-dimension video feature according to the current teaching video of the target teacher based on the improved target end-to-end network, the method further comprises:

[0016] Obtaining a historical single-frame image set, and training an action recognition network according to the historical single-frame image set;

[0017] Obtaining a historical teaching video set, and uniformly cropping the historical teaching video set;

[0018] Dividing the uniformly cropped historical teaching segment set into frames to obtain a historical teaching frame set;

[0019] Recognizing the historical teaching frame set based on the action recognition network, and performing format conversion on the target action of each key frame identified;

[0020] Verifying each format-converted target action;

[0021] When the verification is passed, freezing the parameters of the backbone network through a parameter freezing unit, and training the space adaptation unit and the time sequence adaptation unit according to the historical teaching video set and the each format-converted target action to obtain the improved target end-to-end network.

[0022] In one embodiment, the step of generating multi-window feature groups based on the temporal dimension video features and the spatial dimension video features using a multi-temporal window attention mechanism includes:

[0023] The target frame features are determined based on the temporal and spatial video features.

[0024] The target frame features are scored at the target convolutional layer, and the frame-level scoring results are normalized to obtain the temporal attention weights.

[0025] The target frame features are weighted and summed according to the temporal attention weights to obtain global feature groups;

[0026] By employing a multi-temporal window attention mechanism, multiple local feature groups are generated based on the target frame features, and multi-window feature groups are obtained based on the global feature groups and the multiple local feature groups.

[0027] In one embodiment, the step of fusing the multi-window feature groups using a frame selection network and identifying the atomic actions of the target teacher based on the fusion result includes:

[0028] The multi-window feature groups are aligned.

[0029] The processed multi-window feature groups are subjected to region of interest detection, and the features of the region of interest are determined based on the detection results;

[0030] Generate a target group feature list based on the features of each region of interest;

[0031] The target group feature list is fused using a frame selection network, and the atomic actions of the target teacher are identified based on the fusion result.

[0032] In one embodiment, the step of fusing the target group feature list through a frame selection network and identifying the atomic actions of the target teacher based on the fusion result includes:

[0033] Based on the multi-branch prediction module, each feature of interest in the target group feature list is independently classified and predicted, and the action category prediction score of each group is output through a fully connected layer.

[0034] The action category prediction scores of each group are fused to obtain a stack of action prediction results.

[0035] Obtain the number of groups in the action prediction result stack;

[0036] During the prediction phase, the action prediction results are selected from the grouping number to obtain the maximum prediction score for each category;

[0037] Identify the target teacher's atomic actions based on the maximum predicted score for each category.

[0038] Secondly, this application provides a teacher atomic action recognition device based on an improved end-to-end network, comprising:

[0039] The determination module is used to determine the temporal and spatial video features based on the current teaching video of the target teacher, according to the improved target end-to-end network. The improved target end-to-end network includes a spatial adaptation unit, a temporal adaptation unit, and a parameter freezing unit.

[0040] The generation module is used to generate multi-window feature groups based on the temporal dimension video features and the spatial dimension video features through a multi-temporal window attention mechanism;

[0041] The recognition module is used to fuse the multi-window feature groups through a frame selection network and identify the atomic actions of the target teacher based on the fusion result.

[0042] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation thereof.

[0043] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0044] Fifthly, this application provides a computer program product that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0045] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0046] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art:

[0047] (1) This application introduces Adapter fine-tuning, embedding a spatial adaptation unit (S-Adapter), a temporal adaptation unit (T-Adapter), and a parameter freezing unit in the end-to-end network. The spatial adaptation unit is used to extract local limb features within video frames, the temporal adaptation unit is used to enhance the temporal information of adjacent frames and extract temporal convolutional features, and the parameter freezing unit is used to freeze the parameters of the backbone network. During network training, only the parameters of the above units are trained, and full parameter fine-tuning is not required, thereby reducing graphics card consumption, reducing video memory requirements, and reducing deployment difficulty.

[0048] (2) This application also uses a frame selection network to focus on frames containing effective action information, reducing the interference of background redundancy on model feature learning. Through feature grouping of global window and multiple local windows, global temporal context and local key frame information can be captured respectively, making up for the lack of adaptation of a single window to complex temporal patterns. Independent classification prediction is performed based on the multi-branch prediction module, and the action category prediction score of each group is output through the fully connected layer. At this time, the maximum value is taken according to the category, and the group feature contribution with the most discriminative power for each category can be adaptively selected to avoid weak feature interference and improve the overall classification accuracy, thereby effectively improving the accuracy of atomic action recognition.

[0049] In summary, this application, based on an improved target end-to-end network, determines temporal and spatial video features according to the current teaching video of the target teacher. The improved target end-to-end network includes a spatial adaptation unit, a temporal adaptation unit, and a parameter freezing unit. A multi-temporal window attention mechanism is used to generate multi-window feature groups based on the temporal and spatial video features. A frame selection network is used to fuse these multi-window feature groups, and the atomic actions of the target teacher are identified based on the fusion result. By embedding spatial and temporal adaptation units into the end-to-end network, frame-level temporal and spatial video features are extracted respectively. During network training, only the parameters of these units are trained to reduce GPU consumption. Then, through multi-branch feature processing and adaptive fusion, the ability to discriminate complex temporal information in video action recognition is improved, thereby effectively improving the accuracy and efficiency of atomic action recognition, reducing GPU memory requirements, and lowering deployment difficulty. Attached Figure Description

[0050] Figure 1 This is one of the flowcharts illustrating the teacher atomic action recognition method based on an improved end-to-end network provided in this application embodiment;

[0051] Figure 2 This is a schematic diagram of network training provided in an embodiment of this application;

[0052] Figure 3This is a schematic diagram comparing the average accuracy values ​​provided in the embodiments of this application;

[0053] Figure 4 This is the second flowchart of the teacher atomic action recognition method based on an improved end-to-end network provided in the embodiments of this application;

[0054] Figure 5 This is a schematic diagram of the module structure of the teacher atomic action recognition device based on an improved end-to-end network provided in the embodiments of this application;

[0055] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0058] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0059] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0060] Based on this, embodiments of this application provide a teacher atomic action recognition method based on an improved end-to-end network, referring to... Figure 1 , Figure 1 This is one of the flowcharts illustrating the teacher atomic action recognition method based on an improved end-to-end network provided in this application. In this embodiment, the teacher atomic action recognition method based on an improved end-to-end network includes steps S10 to S30:

[0061] Step S10: Based on the improved target end-to-end network, determine the time dimension video features and spatial dimension video features according to the current teaching video of the target teacher. The improved target end-to-end network includes: a spatial adaptation unit, a temporal adaptation unit, and a parameter freezing unit.

[0062] It should be noted that the improved target end-to-end network, compared to a conventional end-to-end network, improves upon this by embedding spatial adaptation units (S-Adapter), temporal adaptation units (T-Adapter), and parameter freezing units. Each of these units can be created using layered adaptation modules and directly embedded into a specified layer, such as the first 5 layers or layers 6-11. Specifically, the spatial adaptation unit extracts local limb features within video frames, the temporal adaptation unit enhances temporal information from adjacent frames and extracts temporal convolutional features, and the parameter freezing unit freezes the parameters of the backbone network. During network training, GPU consumption is reduced by training only the parameters of the aforementioned units. For the target end-to-end network to be improved, the core points are "backbone preservation + layered adaptation unit embedding + parameter freezing". Backbone preservation refers to preserving the 3DPatch embedding layer and 12 Transformer blocks of the conventional end-to-end network. This conventional end-to-end network can be a VideoMAE network. Since the Transformer block has 12 layers, the spatial adaptation unit can be embedded in the first 5 layers, and the temporal adaptation unit can be embedded in layers 6-11.

[0063] It should be understood that the target teacher can be a lecturer or a teacher trainee. This embodiment does not limit this. For the improved target end-to-end network, after the current lecture video is input, local limb features within the video frame are extracted in the early stage to obtain spatial dimension video features. In the later stage, the temporal information of adjacent frames is enhanced and temporal convolution features are extracted to obtain temporal dimension video features. In the teacher education scenario, the current lecture video can be obtained by a camera set up indoors.

[0064] Furthermore, before step S10, the method includes: acquiring a set of historical single-frame images and training an action recognition network based on the set of historical single-frame images; acquiring a set of historical teaching videos and uniformly cropping the set of historical teaching videos; dividing the uniformly cropped set of historical teaching segments into frames to obtain a set of historical teaching frames; recognizing the set of historical teaching frames based on the action recognition network and converting the target actions of each recognized keyframe into a format; verifying the target actions after each format conversion; and when the verification passes, freezing the parameters of the backbone network through a parameter freezing unit, and training a spatial adaptation unit and a temporal adaptation unit based on the set of historical teaching videos and the target actions after each format conversion to obtain an improved target end-to-end network.

[0065] Understandably, a historical single-frame image set refers to a collection of individual frames gathered at historical moments. Taking a teacher education scenario as an example, following the approach of Atomic Visual Actions (AVA), a total of 9316 target keyframe actions were collected from 100 teaching videos. These actions comprise 10 categories: explaining, nodding, bowing, walking, standing, standing sideways, writing on the blackboard, frequent hand gestures, pointing at the PPT, explaining while facing the PPT, and holding the lecture notes. See Table 1 for details.

[0066] Table 1:

[0067]

[0068] It should be understood that, on the one hand, after obtaining the historical single-frame image set, each single-frame image in the set can be labeled, and then an action recognition network that assists in labeling keyframes can be trained. This action recognition network can be a YOLO network. On the other hand, a historical lecture video set will also be collected. This historical lecture video set can be a collection of lecture videos from multiple teacher trainees and teachers at historical moments. At this time, a series of operations such as uniform cropping and frame segmentation will be performed on the historical lecture video set. For example, it can be uniformly cropped to a 9-minute segment, and then the pre-trained YOLO network is used for action recognition. The target actions of each keyframe are converted into a format. At this time, the target action format of the keyframe can be VIA format, which is convenient for subsequent operations. Then, further filtering and standardization are performed, and a unique and consistent label is assigned to the same target in the video sequence according to the multi-target tracking strategy. This label can be an ID, specifically represented as person_id. Both the historical single-frame image set and the historical lecture video set can be directly collected from the Internet.

[0069] It should also be emphasized that, since spatial adaptation units and temporal adaptation units are embedded during the training of the backbone network in this embodiment, only fine-tuning of the parameters of the spatial and temporal adaptation units is required, without the need for full parameter tuning, thus reducing GPU power consumption. Furthermore, the prerequisite for performing the above fine-tuning is that the parameters of the backbone network need to be frozen. For example, refer to... Figure 2 , Figure 2The diagram illustrates network training. Specifically, boxes A and C represent the backbone network, which represents the frozen parameters, while box B represents the adaptation unit, which represents the training parameters. During training, the parameters of the backbone network do not need to be involved; only the parameters of the spatial and temporal adaptation units are trained. This backbone network includes, but is not limited to, patch embedding, positional encoding, Transformer attention layers, and MLP layers. By training only the parameters of the spatial and temporal adaptation units, the total number of parameters is significantly reduced compared to the full-parameter fine-tuning of existing technologies, and memory consumption is reduced, thereby achieving the goal of reducing memory requirements and deployment difficulty.

[0070] Further, step S10 includes: determining the current image block based on the current teaching video of the target teacher, using the previous conventional unit in the improved target end-to-end network; reshaping the current image block to obtain the current three-dimensional spatial structure image; upsampling the current three-dimensional spatial structure image based on the spatial adaptation unit in the improved target end-to-end network, and extracting local features from the upsampled three-dimensional spatial structure image to obtain local limb features; downsampling the local limb features to obtain spatial dimension video features; upsampling the spatial dimension video features based on the temporal adaptation unit in the improved target end-to-end network, and performing temporal convolution on the upsampled continuous multi-frame three-dimensional features to obtain temporal convolution features; downsampling the temporal convolution features, and performing a linear transformation on the downsampled temporal features to obtain temporal dimension video features.

[0071] It should be understood that the current image patch refers to the image patch output by the previous conventional unit in the improved target end-to-end network based on the current teaching video. This previous conventional unit can be the previous Transformer layer. For the spatial adaptation unit in the improved target end-to-end network, operations include, but are not limited to, upsampling, local feature extraction, and downsampling. Upsampling and downsampling are respectively used through two linear layers. Among them, local limb features include, but are not limited to, subtle features such as hands and limb posture, specifically:

[0072]

[0073] in, Indicates local limb features, LN represents 3D depthwise separable convolution (number of groups = number of input channels), and LN represents layer normalization. Indicates the current image patch. This indicates the kernel size of a 3D depth-separable convolution.

[0074] It is understandable that, for the temporal adaptation unit in the improved target end-to-end network, operations include, but are not limited to, upsampling, temporal convolution, downsampling, and linear transformation, specifically:

[0075]

[0076] in, Representing temporal convolutional features, This represents 1D temporal depthwise convolution, which enhances the temporal correlation of actions. Indicates a linear projection layer. Indicates local limb features, This indicates the kernel size of a 1D temporal depthwise convolution.

[0077] Step S20: Using a multi-temporal window attention mechanism, generate multi-window feature groups based on the temporal dimension video features and the spatial dimension video features.

[0078] Understandably, in order to capture key information at different time scales in the current teaching video, this embodiment will also introduce a multi-time window attention mechanism. At this time, the multi-time window attention mechanism can be used to perform differentiated feature grouping and generate feature groups of multiple local time windows, which is called multi-window feature grouping. Different group features have different discrimination capabilities for different action categories. For example, the global window is more suitable for long-term actions, while the local window is more suitable for short-term actions, thereby making up for the inadequacy of a single window in adapting to complex temporal patterns.

[0079] Step S30: The multi-window feature groups are fused using a frame selection network, and the atomic actions of the target teacher are identified based on the fusion result.

[0080] It should be understood that the frame selection network is used to focus on frames containing effective action information in a fusion manner. Compared with the existing technology, which is subject to interference from complex backgrounds and may result in misidentification, it can reduce the interference of background redundancy on model feature learning and solve the defects of redundant background frames in full frame input interfering with feature extraction and affecting recognition accuracy. In this embodiment, the fusion result of multi-window feature grouping is used to identify the atomic actions of the target teacher. After solving the above defects, it can effectively improve the accuracy and efficiency of atomic action recognition.

[0081] Further, step S30 includes: aligning the multi-window feature groups; detecting regions of interest (ROIs) on the processed multi-window feature groups and determining the features of the ROIs based on the detection results; generating a target group feature list based on the features of each ROI; fusing the target group feature list through a frame selection network and identifying the atomic actions of the target teacher based on the fusion result.

[0082] It can be understood that in order to achieve feature grouping of the region of interest (ROI), after obtaining the multi-window feature grouping, operations such as ROI alignment and ROI detection need to be performed on each window feature grouping, and the features of the ROI are extracted. The ROI can be a human body box. At this time, the target grouping feature list can be generated according to the features of each ROI. The target grouping feature list can be expressed as group_roi_feats_list, and the shape of each element is [R, M, 1, Ph, Pw], where R represents the number of ROIs, M represents the number of feature channels, Ph represents the height after pooling, and Pw represents the width after pooling.

[0083] Further, the step of fusing the target grouping feature list through the frame selection network and identifying the atomic actions of the target teacher according to the fusion result includes: based on the multi-branch prediction module, independently classifying and predicting each interesting feature in the target grouping feature list, and outputting the action category prediction scores of each group through the fully connected layer; fusing the action category prediction scores of each group to obtain an action prediction result stack; obtaining the number of groups of the action prediction result stack; in the prediction stage, selecting from the action prediction result stack according to the number of groups to obtain the maximum prediction scores of each category; identifying the atomic actions of the target teacher according to the maximum prediction scores of each category.

[0084] It should be understood that in order to strengthen the contribution of discriminative features, avoid the interference of weak features, and improve the overall classification accuracy, this application also introduces a multi-branch prediction module to independently classify and predict each interesting feature in the target grouping feature list. The multi-branch prediction module can be a multi-group bounding box head (MultiGroupBBoxHead), and then the action category prediction scores of each group are output through the fully connected layer, and an action prediction result stack is obtained through the fusion method. The action prediction result stack can be a logits_stack with the shape of [G, R, C], where G represents the number of groups and C represents the number of categories.

[0085] It should also be emphasized that in the prediction stage, the action prediction result stack is processed along the grouping dimension, and for each category c (0 ≤ c < C) and each ROI m (0 ≤ r < R), the maximum score of this category in all groups is selected as the final prediction result. Among them, the operation of taking the maximum score by category can be expressed as:

[0086]

[0087] Among them, represents the predicted score after fusion, represents the total number of feature groupings, This represents the total number of ROIs. Indicates the total number of action categories. Indicates the first In the feature grouping, the first The corresponding ROI is the Predicted scores for each category This indicates all groups Take the maximum value.

[0088] It should be noted that the above formula means that for each region of interest and each action category, the maximum predicted score from all feature groups is selected as the final predicted score for that region in that category. To reduce confusion between similar action categories, this embodiment significantly reduces the confusion rate by strengthening the category-specific optimal features for easily confused action categories. Specifically, the differences between similar actions are only reflected in local features. For example, the "dribble action" in "basketball" depends on a local feature group, while the "passing action" in "volleyball" depends on another local feature group. Taking the maximum value by category can capture these subtle differences, while commonly used averaging or weighted fusion will blur the category-specific features, resulting in insufficient capture of differences. At this time, the atomic actions of the target teacher can be identified based on the maximum predicted score of each category, thereby effectively improving the accuracy and efficiency of atomic action identification.

[0089] It should be understood that, reference Figure 3 , Figure 3 The diagram illustrates the comparison of average precision. Specifically, using a conventional end-to-end network and an improved target end-to-end network as examples, when both backbone networks are VIT-B and require additional labels, the average precision of the conventional end-to-end network is 31.8, while the average precision of the improved target end-to-end network is 35.4. It is evident that the average precision of the conventional end-to-end network is significantly lower than that of the improved target end-to-end network in this embodiment. When both backbone networks are VIT-L and require additional labels, the average precision of the conventional end-to-end network is 37.0, while the average precision of the improved target end-to-end network is 39.1. Again, the average precision of the conventional end-to-end network is significantly lower than that of the improved target end-to-end network in this embodiment. In other words, this embodiment uses an end-to-end network embedded with spatial adaptation units, timing adaptation units, and parameter freezing units, significantly improving the average precision, and the improved target end-to-end network exhibits better performance.

[0090] This embodiment is based on an improved target end-to-end network. It determines temporal and spatial video features based on the target teacher's current teaching video. The improved target end-to-end network includes a spatial adaptation unit, a temporal adaptation unit, and a parameter freezing unit. A multi-temporal window attention mechanism is used to generate multi-window feature groups based on the temporal and spatial video features. A frame selection network is then used to fuse these multi-window feature groups, and the atomic actions of the target teacher are identified based on the fusion result. By embedding spatial and temporal adaptation units into the end-to-end network, frame-level temporal and spatial video features are extracted, respectively. During network training, only the parameters of these units are trained to reduce GPU consumption. Then, through multi-branch feature processing and adaptive fusion, the ability to discriminate complex temporal information in video action recognition is improved. This effectively improves the accuracy and efficiency of atomic action recognition, reduces GPU memory requirements, and enhances the user experience.

[0091] In one specific implementation, this application provides steps for determining multi-window feature groups. Please refer to... Figure 4 , Figure 4 This is the second flowchart illustrating the teacher atomic action recognition method based on an improved end-to-end network provided in this application embodiment. Step S20 includes steps S201 to S204:

[0092] Step S201: Determine the target frame features based on the time dimension video features and the spatial dimension video features.

[0093] It should be noted that for both temporal and spatial video features, the shape is [B, M, T, H, W], where B represents the batch size, M represents the number of feature channels, T represents the number of time frames, H represents the height, and W represents the width. In this case, local average pooling can be used to compress the spatial dimension to obtain the target frame features, specifically:

[0094]

[0095] in, This represents the features of the target frame.

[0096] Step S202: Frame-level scoring is performed on the target frame features based on the target convolutional layer, and the frame-level scoring results are normalized to obtain the temporal attention weights.

[0097] It is understandable that after obtaining the target frame features, the target frame features can be scored at the frame level based on the target convolutional layer. This target convolutional layer can be a 1D convolutional layer. At this time, the score can be converted into temporal attention weights through normalization. That is, the higher the score of the frame, the greater the temporal attention weight, thus achieving "dynamic focusing". The sum of the temporal attention weights of all frames can be 1.

[0098] Step S203: The target frame features are weighted and summed according to the temporal attention weights to obtain global feature groups.

[0099] It should be understood that after obtaining the temporal attention weights, the target frame features can be weighted and summed according to the temporal attention weights to achieve global temporal weighted pooling.

[0100] Step S204: Using a multi-temporal window attention mechanism, generate multiple local feature groups based on the target frame features, and obtain multi-window feature groups based on the global feature groups and the multiple local feature groups.

[0101] It is understandable that after obtaining the target frame features, this embodiment will also configure multi-window parameters through a multi-time window attention mechanism and inject Gaussian prior into the attention weight calculation. Specifically, with the middle frame of the video as the center, different window sizes correspond to different Gaussian variances, generating multiple local feature groups. For example, windows with sizes of 2, 4, and 6 are windows with time steps.

[0102] This embodiment determines target frame features based on the temporal and spatial video features; performs frame-level scoring on the target frame features using the target convolutional layer, and normalizes the frame-level scoring results to obtain temporal attention weights; performs weighted summation on the target frame features based on the temporal attention weights to obtain global feature groups; and generates multiple local feature groups based on the target frame features using a multi-temporal window attention mechanism, and obtains multi-window feature groups based on the global feature groups and the multiple local feature groups. Through this method, after determining the target frame features based on the temporal and spatial video features, on the one hand, global feature groups are determined using global temporal weighted pooling; on the other hand, multiple local feature groups are generated based on the target frame features. Then, by combining the global feature groups and the multiple local feature groups, global temporal context and local keyframe information are captured through the feature groups of the global window and the multiple local windows, respectively, thereby effectively improving the accuracy of obtaining multi-window feature groups.

[0103] The teacher atomic action recognition device based on an improved end-to-end network provided in this application is described below. The teacher atomic action recognition device based on an improved end-to-end network described below can be referred to in conjunction with the teacher atomic action recognition method based on an improved end-to-end network described above. Please refer to... Figure 5 , Figure 5 This is a schematic diagram of the module structure of the teacher atomic action recognition device based on an improved end-to-end network provided in this application embodiment, including:

[0104] The determination module T10 is used to determine the temporal and spatial video features based on the current teaching video of the target teacher, according to the improved target end-to-end network. The improved target end-to-end network includes a spatial adaptation unit, a temporal adaptation unit, and a parameter freezing unit.

[0105] The generation module T20 is used to generate multi-window feature groups based on the time-dimensional video features and the spatial-dimensional video features through a multi-temporal window attention mechanism.

[0106] The recognition module T30 is used to fuse the multi-window feature groups through a frame selection network and recognize the atomic actions of the target teacher based on the fusion result.

[0107] This embodiment is based on an improved target end-to-end network. It determines temporal and spatial video features based on the target teacher's current teaching video. The improved target end-to-end network includes a spatial adaptation unit, a temporal adaptation unit, and a parameter freezing unit. A multi-temporal window attention mechanism is used to generate multi-window feature groups based on the temporal and spatial video features. A frame selection network is then used to fuse these multi-window feature groups, and the atomic actions of the target teacher are identified based on the fusion result. By embedding spatial and temporal adaptation units into the end-to-end network, frame-level temporal and spatial video features are extracted, respectively. During network training, only the parameters of these units are trained to reduce GPU consumption. Then, through multi-branch feature processing and adaptive fusion, the ability to discriminate complex temporal information in video action recognition is improved. This effectively improves the accuracy and efficiency of atomic action recognition, reduces GPU memory requirements, and enhances the user experience.

[0108] It is understood that the detailed functional implementation of each of the above modules can be found in the description of the aforementioned method embodiments, and will not be repeated here.

[0109] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0110] Based on the methods in the above embodiments, this application provides an electronic device, please refer to... Figure 6 , Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0111] It should be noted that the system may include: a processor 10, a communications interface 20, a memory 30, and a communication bus 40. The processor 10, communications interface 20, and memory 30 communicate with each other via the communication bus 40. The processor 10 can invoke logical instructions stored in the memory 30 to execute the methods described in the above embodiments.

[0112] Furthermore, the logical instructions in the aforementioned memory 30 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0113] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0114] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0115] It is understood that the processor in the embodiments of this application can be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, field-programmable gate arrays, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0116] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory, flash memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, hard disks, portable hard disks, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor.

[0117] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. Those skilled in the art will readily understand that the above descriptions are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A teacher atomic action recognition method based on an improved end-to-end network, characterized in that, include: Based on the improved target end-to-end network, temporal and spatial video features are determined according to the current teaching video of the target teacher. The improved target end-to-end network includes a spatial adaptation unit, a temporal adaptation unit, and a parameter freezing unit. A multi-temporal window attention mechanism is used to generate multi-window feature groups based on the temporal dimension video features and the spatial dimension video features. The multi-window feature groups are fused using a frame selection network, and the atomic actions of the target teacher are identified based on the fusion results. The step of generating multi-window feature groups based on the temporal dimension video features and the spatial dimension video features using a multi-temporal window attention mechanism includes: The target frame features are determined based on the temporal and spatial video features. The target frame features are scored at the target convolutional layer, and the frame-level scoring results are normalized to obtain the temporal attention weights. The target frame features are weighted and summed according to the temporal attention weights to obtain global feature groups; By employing a multi-temporal window attention mechanism, multiple local feature groups are generated based on the target frame features, and multi-window feature groups are obtained based on the global feature groups and the multiple local feature groups.

2. The method as described in claim 1, characterized in that, The steps of determining temporal and spatial video features based on the target teacher's current teaching video using the improved target end-to-end network include: Based on the previous conventional unit in the improved target end-to-end network, the current image block is determined according to the target teacher's current teaching video; The current image block is reshaped to obtain the current three-dimensional spatial structure image; The current three-dimensional spatial structure image is upsampled based on the spatial adaptation unit in the improved target end-to-end network, and local features are extracted from the upsampled three-dimensional spatial structure image to obtain local limb features. The local limb features are downsampled to obtain spatial dimension video features; The spatial dimension video features are upsampled based on the temporal adaptation unit in the improved target end-to-end network, and the upsampled three-dimensional features of multiple consecutive frames are temporally convolved to obtain temporal convolution features. The temporal convolutional features are downsampled, and the downsampled temporal features are linearly transformed to obtain the temporal dimension video features.

3. The method as described in claim 1, characterized in that, Before the step of determining the temporal and spatial video features based on the target teacher's current teaching video using the improved target end-to-end network, the method further includes: Acquire a set of historical single-frame images, and train an action recognition network based on the set of historical single-frame images; Obtain the historical lecture video collection and uniformly crop the historical lecture video collection; The uniformly cropped set of historical lecture segments is divided into frames to obtain a set of historical lecture frames. The historical teaching frame set is identified based on the action recognition network, and the target action of each identified key frame is converted into a format. Verify the target action after each format conversion; When the verification is successful, the parameters of the backbone network are frozen by the parameter freezing unit, and the improved target end-to-end network is obtained based on the historical teaching video set and the target action training space adaptation unit and temporal adaptation unit after conversion of each format.

4. The method according to any one of claims 1 to 3, characterized in that, The step of fusing the multi-window feature groups using a frame selection network and identifying the atomic actions of the target teacher based on the fusion result includes: The multi-window feature groups are aligned. The processed multi-window feature groups are subjected to region of interest detection, and the features of the region of interest are determined based on the detection results; Generate a target group feature list based on the features of each region of interest; The target group feature list is fused using a frame selection network, and the atomic actions of the target teacher are identified based on the fusion result.

5. The method as described in claim 4, characterized in that, The step of fusing the target group feature list through a frame selection network and identifying the atomic actions of the target teacher based on the fusion result includes: Based on the multi-branch prediction module, each feature of interest in the target group feature list is independently classified and predicted, and the action category prediction score of each group is output through a fully connected layer. The action category prediction scores of each group are fused to obtain a stack of action prediction results. Obtain the number of groups in the action prediction result stack; During the prediction phase, the action prediction results are selected from the grouping number to obtain the maximum prediction score for each category; Identify the target teacher's atomic actions based on the maximum predicted score for each category.

6. A teacher atomic action recognition device based on an improved end-to-end network, characterized in that, include: The determination module is used to determine the temporal and spatial video features based on the current teaching video of the target teacher, according to the improved target end-to-end network. The improved target end-to-end network includes a spatial adaptation unit, a temporal adaptation unit, and a parameter freezing unit. The generation module is used to generate multi-window feature groups based on the temporal dimension video features and the spatial dimension video features through a multi-temporal window attention mechanism; The recognition module is used to fuse the multi-window feature groups through a frame selection network and recognize the atomic actions of the target teacher based on the fusion result; The generation module is further configured to: determine target frame features based on the temporal video features and the spatial video features; perform frame-level scoring on the target frame features based on the target convolutional layer, and normalize the frame-level scoring results to obtain temporal attention weights; perform weighted summation on the target frame features based on the temporal attention weights to obtain global feature groups; and generate multiple local feature groups based on the target frame features through a multi-temporal window attention mechanism, and obtain multi-window feature groups based on the global feature groups and the multiple local feature groups.

7. An electronic device, characterized in that, include: At least one memory for storing computer programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method as described in any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is run on the processor, it causes the processor to perform the method as described in any one of claims 1-5.

9. A computer program product, characterized in that, When the computer program product is run on a processor, the processor causes the processor to perform the method as described in any one of claims 1-5.

Images