A method and apparatus for action recognition
By using Gaussian distribution test and action type frequency verification in the video action recognition model, the problem of inaccurate action recognition in the prior art is solved, and higher recognition accuracy and robustness are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- KWEICHOW MOUTAI COMPANY
- Filing Date
- 2023-05-26
- Publication Date
- 2026-07-24
Smart Images
Figure CN116778574B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to an action recognition method and apparatus. Background Technology
[0002] Video motion recognition technology has a wide range of applications in industrial production, agricultural production, and daily life. For example, in industrial production, real-time motion recognition of normally operating machinery and equipment can capture abnormal movements, which can promptly identify potential production risks and reduce corporate losses. In daily life, real-time fall recognition of elderly people living alone and reporting it to the police can greatly reduce the risk to their lives.
[0003] Currently, action recognition mainly uses deep learning methods, which utilize network models to extract the depth and spatiotemporal features of video frame images, and then classify the action categories based on the depth and spatiotemporal features. At present, the action category classification of the main neural network models is based on confidence, which has the disadvantage of inaccuracy in the classification results. Summary of the Invention
[0004] Based on this, an action recognition method and apparatus are provided to improve the problem of inaccurate action recognition in the prior art.
[0005] On the one hand, an action recognition method is provided, the method comprising: Obtain a first frame sequence, the first frame sequence including L frames of the first video segment to be identified; The first frame sequence is input into the action recognition model to obtain the first action type sequence of the video frame and the first Gaussian regression sequence predicted for each action type, wherein the first action type sequence includes the action type predicted for each video frame. The target action type is determined based on the frequency of action types in the first action type sequence; A Gaussian distribution test is performed on the first Gaussian regression sequence corresponding to the target action type to obtain a test value. If the test value is greater than the test threshold, the target action type is determined to be the action recognition result of the first video segment.
[0006] In one embodiment, after obtaining the first action type sequence of the video frame and the first Gaussian regression sequence predicted for each action type, the method further includes: An action type combination sequence is obtained based on a first action type sequence and a second action type sequence. The second action type sequence is obtained based on a second frame sequence, which includes multiple video frames of a second video segment that are continuous with the first video segment. Determining the target action type based on the frequency of action types in the first action type sequence includes: The target action type is determined based on the frequency of the action type in the action type combination sequence.
[0007] In one embodiment, the action type combination sequence includes a first action type portion and a second action type; obtaining the action type combination sequence based on the first action type sequence and the second action type sequence includes: The second action type portion is updated according to the first action type sequence to obtain the updated second action type portion; The first action type part is updated based on the second action type part before the update to obtain the updated first action type part, so as to obtain the updated action type combination sequence.
[0008] In one embodiment, after obtaining the first action type sequence of the video frame and the first Gaussian regression sequence predicted for each action type, the method further includes: A Gaussian regression combination sequence is obtained based on the first Gaussian regression sequence and the second Gaussian regression sequence corresponding to each action type, wherein the second Gaussian regression sequence is obtained based on the second frame sequence; The Gaussian distribution test based on the first Gaussian regression sequence corresponding to the target action type includes: A Gaussian distribution test is performed on the Gaussian regression combination sequence corresponding to the target action type to obtain the test value.
[0009] In one embodiment, the Gaussian regression combination sequence includes a first Gaussian regression part and a second Gaussian regression part; The step of obtaining a Gaussian regression combination sequence based on the first Gaussian regression sequence and the second Gaussian regression sequence corresponding to each action type includes: The second Gaussian regression part is updated based on the first Gaussian regression sequence corresponding to the action type, resulting in the updated second Gaussian regression part. The first Gaussian regression part is updated based on the second Gaussian regression part before the update to obtain the updated Gaussian regression combination sequence.
[0010] In one embodiment, determining the target action type based on the frequency of action types in the first action type sequence includes: The confidence level of each action type is obtained based on its frequency in the first action type sequence. The action type with the highest confidence level is identified as the target action type.
[0011] In one embodiment, before determining the action type with the highest confidence level as the target action type, the method further includes: Determine whether the highest confidence level is greater than the confidence threshold. If so, determine the predicted action type with the highest confidence level as the target action type.
[0012] In one embodiment, the action recognition module is trained according to the following steps: Obtain a frame sequence sample set, which includes L training video frames, wherein the training video frames include action type label values and Gaussian distribution annotation values; Input the frame sequence sample set into the initial model to obtain the predicted action type sequence for each training video frame and the predicted Gaussian regression sequence corresponding to each action type; The initial model is iteratively trained based on the action recognition loss to obtain an action recognition model. The action recognition loss includes action type loss and Gaussian regression loss. The action type loss is obtained based on the action type label value and the predicted action type sequence, and the Gaussian regression loss is obtained based on the Gaussian distribution label value and the predicted Gaussian regression sequence.
[0013] In one embodiment, the Gaussian distribution label value is determined according to the following mathematical expression: ; in, For the first Gaussian distribution annotation values of training video frames. To train the keyframe index in the video frames, For the starting frame index, To terminate the frame index, .
[0014] On the other hand, an action recognition device is provided, the device comprising: The video frame acquisition module is used to acquire a first frame sequence, the first frame sequence including L video frames of the first video segment to be identified; The model inference module is used to obtain a first action type sequence of the video frame and a first Gaussian regression sequence predicted for each action type after the action recognition model is input into the first frame sequence, wherein the first action type sequence includes the action type predicted for each video frame. An action type determination module is used to determine the target action type based on the frequency of action types in the first action type sequence; The verification module is used to perform Gaussian distribution verification based on the first Gaussian regression sequence corresponding to the target action type to obtain a verification value, and if the verification value is greater than the verification threshold, determine that the target action type is the action recognition result of the first video segment.
[0015] The aforementioned action recognition method and apparatus obtain an action type sequence by predicting action types, and determine the possible target action types in the video segment based on the frequency magnitude from the action type sequence. On the other hand, it is assumed that the action follows a Gaussian distribution, and Gaussian regression prediction is performed on the action type. Gaussian distribution test is performed on the target action type. If the Gaussian distribution test value corresponding to the target action type is greater than the test threshold, it is determined that the target action type follows a Gaussian distribution, and the target action type is taken as the recognition result. The method of this application improves the accuracy of action recognition through Gaussian distribution test. Attached Figure Description
[0016] Figure 1 This is a flowchart illustrating an action recognition method in one embodiment; Figure 2 This is a schematic diagram of an action recognition module in one embodiment; Figure 3 This is a schematic diagram of the single-frame classification head output result in one embodiment; Figure 4 This is a schematic diagram of the first frame sequence and the second frame sequence in one embodiment; Figure 5 This is a schematic diagram illustrating the update of an action type combination sequence in one embodiment; Figure 6 This is a flowchart illustrating the action recognition method in another embodiment; Figure 7 This is a structural block diagram of an action recognition device in one embodiment. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0018] Action recognition has attracted much attention due to its potential applications in automated video analysis, video surveillance, sports event analysis, and virtual reality. As research progresses, increasingly higher demands are being placed on action recognition.
[0019] Deep learning has achieved results that almost surpass those of humans in computer vision tasks such as image recognition and object detection. Deep learning methods are also increasingly being used in video tasks. For example, in video-based action recognition, a common approach is to segment video segments into continuous video frames containing temporal information, input the video frames into a pre-trained neural network model to obtain the probability of each possible action type, i.e., the confidence level, and use the action type predicted with the highest confidence level as the final recognition result. However, this approach has a high error rate.
[0020] This application provides an action recognition method that improves the accuracy of action recognition by performing a Gaussian distribution test on the predicted possible action types and taking the action types that conform to the Gaussian distribution as the action recognition result.
[0021] In one embodiment, the action recognition method is as follows: Figure 1 As shown, it includes the following steps: Step 101: Obtain the first frame sequence, which includes L-frame video frames of the first video segment to be identified.
[0022] For example, the video to be identified is segmented into frames to obtain a continuous sequence of frame images. The first video segment is part or all of the video to be identified, and it is correspondingly divided into L frames, where L can be understood as the length of the video frame, and its value is selected according to the action recognition model used.
[0023] Step 102: Input the first frame sequence into the action recognition model, which has been pre-trained.
[0024] As exemplarily illustrated, the action recognition model is as follows: Figure 2 As shown, it includes at least a single-frame classification head for outputting action types and a Gaussian regression head for outputting Gaussian regression results. In particular, it may also include a spatiotemporal feature extraction backbone for action feature extraction. The spatiotemporal feature extraction backbone takes video frames of length L as input. During training, a series of video samples containing different action categories are collected and framed to obtain a frame sequence sample set. The frame sequence sample set is input into the spatiotemporal feature extraction backbone of the initial model to extract the depth spatiotemporal features of the video frames. Commonly used spatiotemporal feature extraction backbones include 3D convolutional neural networks, two-stream networks, and Video Swin Transformers. For scenarios with high real-time requirements, lightweight backbones such as R(2+1)D and TSM can also be selected. The method in this application exemplarily uses ResNet50-3D as the spatiotemporal feature extraction backbone network.
[0025] The output of the spatiotemporal feature extraction backbone is used as the input of the single-frame classification head. The single-frame classification head predicts the action category of a single frame based on the depth features extracted by the backbone network and outputs the prediction result for each frame.
[0026] Specifically, the training video frames in the frame sequence sample set have action type label values. For example, for video samples containing three action categories, "background", "grab", and "release", "background" is regarded as a special action type with a label value of 0; the label value of "release" is determined to be 1; and the label value of "grab" is determined to be "2". When more action types exist, they can be labeled according to 0, 1, ..., C-1, where C is the number of action types.
[0027] The labeling process involves assigning the same label value to all video frames within the duration of an action. For example, assuming the keyframe index is μ, the corresponding action category is c, and the action coverage length is 2t+1, then the single-frame action category label value associated with keyframe μ can be represented as: ; It is understandable that in the mathematical expression of the above single-frame action category label value, the index is... The video frame is the starting frame of the action, with the index being... The video frame is the termination frame of the action.
[0028] After the frame sequence sample set is input into the initial model, the predicted action type for each training video frame is obtained through a single-frame classification head, such as... Figure 3 As shown, the output of the single-frame classification head is of dimension 1. The recognition results for each frame can include the predicted probability of each action type, and the action type with the highest probability is output as the prediction result for that frame. During the training phase, the multi-class cross-entropy loss function is used as the loss function for this branch. The action type loss is obtained based on the action type label value and the predicted action type sequence, and iterative training is performed using this loss.
[0029] The output of the spatiotemporal feature extraction backbone is also used as the input to a single Gaussian regression head. The multi-channel Gaussian regression head performs single-frame Gaussian regression value prediction based on the depth features extracted by the spatiotemporal feature extraction backbone. The range of Gaussian regression values is... The output dimension of this Gaussian regression head is That is, each action category channel outputs an L-dimensional Gaussian regression vector. During the training phase, the sigmoid cross-entropy loss function is used as the loss function for this branch. The Gaussian regression loss is obtained based on the Gaussian distribution label value and the predicted Gaussian regression sequence, and iterative training is performed based on this.
[0030] It is understandable that the Gaussian distribution label value for each frame can be determined according to the following mathematical expression: ; in, For the first Gaussian distribution annotation values of training video frames. .
[0031] Understandably, the Gaussian distribution annotation process described above treats a complete action as following a Gaussian distribution based on the Gaussian modeling method. A complete temporal action is described using a Gaussian distribution, with the start point, peak point, and end point corresponding to the left and right edges of the Gaussian distribution, respectively. quantile, mean quantile, and right quantile The length of the quantiles varies dynamically depending on the action type, and the Gaussian distribution is also different. This solves the problems of insufficient action information extraction due to too short a number of frames and the introduction of additional background information due to too long a number of frames when modeling based on fixed-length frame sequence segments.
[0032] The iterative process includes training the model using stochastic gradient descent until the convergence condition is met. For example, after each training round, the model is validated on the test set, the mean square error of the multi-channel Gaussian regression values is calculated, and the model with the smallest mean square error on the test set in all rounds is saved to disk for inference.
[0033] Using the above training process, the model has a single-frame classification head and a multi-channel Gaussian regression head. It predicts the action type for each frame of the image and outputs the Gaussian regression value for each frame. The strong supervision information of the single-frame classification head can accelerate the convergence speed of the multi-channel Gaussian regression head and enhance the generalization ability of the network model.
[0034] The action recognition model obtained using the above method is used for action recognition. After inputting the action recognition model into the first frame sequence to be recognized, the first action type sequence of the video frame is obtained. And the first Gaussian regression sequence predicted for each action type The first action type sequence includes the action type predicted for each video frame. It is still represented by values in the form of 0, 1, ..., C-1. The first Gaussian regression sequence includes action types. The corresponding Gaussian regression value predicted for each video frame .
[0035] Step 103: Determine the target action type based on the frequency of action types in the first action type sequence.
[0036] It is understandable that the target action type determined in step 103 is a non-background action. Specifically, the frequency of non-zero categories in the first action type sequence is counted, assuming that the frequency of each action type is represented as... The frequencies of each action type are: ; The frequency of action types is the confidence level of the predicted action. In one embodiment of this example, the action type with the highest confidence level S is determined as the target action type.
[0037] Step 104: Perform Gaussian distribution test on the first Gaussian regression sequence corresponding to the target action type to obtain the test value. If the test value is greater than the test threshold, determine that the target action type is the action recognition result of the first video segment.
[0038] As can be illustrated, the Gaussian distribution test can be... The test is performed on the first Gaussian regression sequence corresponding to the target action type. The test verifies whether the predicted action type follows a Gaussian distribution. If it does not follow a Gaussian distribution, it can still be considered that there is no action in the current video segment.
[0039] Record Gaussian regression for non-background action types The p-value of the test is expressed as , The test threshold is , Test threshold For example, it is 0.05, if the action category c corresponds to Test p-value If the regression prediction value corresponding to action category c satisfies a Gaussian distribution, then the predicted action category is output as c.
[0040] The action recognition method adopted in this application is based on Gaussian distribution testing to verify the predicted action type. If the predicted action type does not meet the Gaussian distribution, the prediction result is excluded, and only the action type that meets the Gaussian distribution is output, thereby improving the accuracy of action recognition. On the other hand, this application takes a probabilistic and statistical approach, using the overall regression prediction value as the test object, without relying on the prediction result of a single frame. This allows for a larger deviation in the regression of a single frame, further improving the accuracy and robustness of the prediction.
[0041] As one implementation of the above embodiments, the action recognition method further includes a confidence threshold test. This means calculating the highest confidence S in the first action type sequence output by the action recognition model, and recording the confidence threshold. , For example, it is 0.8, if This indicates that no action has occurred in the current video segment. Then, we further examine whether the first Gaussian regression sequence corresponding to the action type with the highest confidence S satisfies a Gaussian distribution.
[0042] In real-world motion recognition scenarios, an action may be fully contained within a video frame sequence of length L, or it may be segmented at the beginning or end of the frame sequence, leading to inaccurate or even missed actions.
[0043] The action recognition method provided in this application, in order to reduce the inaccuracy of action recognition caused by the segmentation of complete actions, performs overall statistical modeling of more than L predicted action types and Gaussian regression values.
[0044] Specifically, on the one hand, by recognizing the L1 frame video frame in the first video segment through the action recognition model, a first action type sequence including L1 output is obtained. On the other hand, by recognizing the L2 frame video frame in the second video segment, a second action type sequence including L2 output is obtained. The first action type sequence and the second action type sequence are combined to obtain an action type combination sequence including L1+L2 output.
[0045] The first video segment and the second video segment are two consecutive segments in the same video to be identified. The second video segment can be before or after the first video segment, and the second video segment can be divided into more or fewer video frames than the first video segment. Action recognition is performed using the same or different action recognition model as the first video segment.
[0046] Probability statistics are performed on the action types in the action type combination sequence to determine the target action type.
[0047] The above process can avoid inaccurate identification of target action type caused by action segmentation.
[0048] In one embodiment of the above examples, Gaussian regression value prediction for each action type in the second video segment is also included.
[0049] Specifically, a first frame sequence containing L1 video frames is obtained through frame segmentation of the first video segment. This sequence is then input into the action recognition model to obtain the first Gaussian regression sequence corresponding to each action type in the first frame sequence. The first Gaussian regression sequence contains an L1-dimensional Gaussian regression sequence (or vector). A second frame sequence containing L2 video frames is obtained through frame segmentation of the second video segment. This sequence is then input into the action recognition model to obtain the second Gaussian regression sequence corresponding to each action type in the second frame sequence. The second Gaussian regression sequence contains an L2-dimensional Gaussian regression sequence (or vector). The first and second Gaussian regression sequences are combined to obtain an L1+L2 dimensional Gaussian regression combination sequence. A Gaussian distribution test is then performed on the Gaussian regression combination sequence corresponding to the target action type to obtain a test value.
[0050] The above process can avoid inaccurate Gaussian distribution tests caused by segmenting actions.
[0051] As one implementation of the above embodiment, the combination of the Gaussian regression sequence and the action type sequence of the second frame sequence are implemented simultaneously, and the second frame sequence and the first frame sequence are processed and recognized using the same action recognition model. Therefore, the second frame sequence and the first frame sequence have the same length L.
[0052] In one embodiment, such as Figure 4As shown. The second frame sequence is the previous input to the action recognition model, and the first frame sequence is the current input to the action recognition model. An action type buffer sequence is configured to combine the first action type sequence and the second action type sequence. Configured to include the first action type portion Second Action Type Part The first action type part is used to record the second action type sequence, and the first part is used to record the first action type. Specifically, each time the action recognition model obtains a new action type output, the first action type part is replaced with the second action type part before the update, based on the currently output first action type sequence. Replace the second action type part, such as Figure 5 As shown. The update method is specifically described as follows: ; The updated action type buffer sequence is the current output action type combination sequence.
[0053] Gaussian regression combined sequences are obtained in a similar manner, as explained below: Configure Gaussian regression buffer sequence for action type j The Gaussian regression buffer sequence includes the first Gaussian regression part. Second Gaussian regression part After each input action recognition model obtains a new Gaussian regression output, the first Gaussian regression part is updated based on the second Gaussian regression part before the update, and the first Gaussian regression sequence is updated based on the current output. The second Gaussian regression part is updated, and the update method can be described as follows: ; Understandably, by using a configuration buffer sequence, combining the previous output with the current output, and performing action type identification and Gaussian distribution verification on 2L frames for each inference, the inaccurate identification caused by the segmentation of complete actions can be avoided.
[0054] like Figure 6 As shown, an action recognition method is provided. Given an offline video or a real-time video stream, an offline-trained action recognition convolutional neural network model is used for result inference. In the process of model inference, in order to reduce the inaccuracy of single-frame Gaussian regression values caused by the segmentation of complete actions, 2L Gaussian regression values are statistically modeled as a whole.
[0055] Configure action type buffer sequence or vector: ; Configure the Gaussian regression buffer matrix: ; In the Gaussian regression buffer matrix, the i-th row represents the Gaussian regression sequence corresponding to action category i.
[0056] Select a series of consecutive video frames of length L, and scale each of the video frames to a size of 112 x 112. Secondly, the scaled frame sequence is input into the action recognition model, and the model is updated based on the output. and Specifically, the action category results identified by the current model are as follows: The update process of the action type buffer sequence F is as follows: ; The Gaussian regression prediction value identified by the current model is The update process of the Gaussian regression buffer matrix G is as follows: ; Then, based on the updated buffer result F, predict the action type and its confidence level. Specifically, count the frequency of non-zero categories in the action type buffer sequence F, and express the frequency of each action type as follows: Then the frequency, i.e. the confidence level, of each action type is: ; The action type output by the single-frame classification head is: ; Let S be the confidence score of action type c, and then perform a test on the non-background category rows in the Gaussian regression buffer matrix G. The test verifies whether the Gaussian regression predictions satisfy a Gaussian distribution. In this embodiment, the action classification confidence threshold is used. , Test threshold If the confidence level of the predicted action If it is true, it indicates that no action has occurred in the current time segment; otherwise, if the action type c corresponds to... Test p-value If the regression prediction value for action category c does not satisfy a Gaussian distribution, then there is no action in the current time segment; otherwise, the predicted action category is output as c.
[0057] It should be understood that, although Figure 1 , Figure 6 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 , Figure 6At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0058] In one embodiment, such as Figure 7 As shown, an action recognition device is provided, including: a video frame acquisition module 201, a model inference module 202, an action type determination module 203, and a verification module 204, wherein: The video frame acquisition module 201 is used to acquire a first frame sequence, the first frame sequence including L frames of the first video segment to be identified; The model inference module 202 is used to obtain a first action type sequence of the video frame and a first Gaussian regression sequence predicted for each action type after inputting the action recognition model into the first frame sequence, wherein the first action type sequence includes the action type predicted for each video frame. Action type determination module 203 is used to determine the target action type based on the frequency of action types in the first action type sequence; The verification module 204 is used to perform Gaussian distribution verification based on the first Gaussian regression sequence corresponding to the target action type to obtain a verification value, and if the verification value is greater than the verification threshold, determine that the target action type is the action recognition result of the first video segment.
[0059] By predicting action types, an action type sequence is obtained, and the target action type that may exist in the video segment is determined from the action type sequence based on the frequency magnitude. On the other hand, it is assumed that the action follows a Gaussian distribution, so Gaussian regression is performed to predict the action type, and Gaussian distribution test is performed on the target action type. If the Gaussian distribution test value corresponding to the target action type is greater than the test threshold, it is determined that the target action type follows a Gaussian distribution, and the target action type is taken as the recognition result, which improves the accuracy of action recognition.
[0060] In another embodiment, the action type determination module 203 obtains the confidence level of each action type based on the frequency of the action type in the first action type sequence; and determines the action type with the highest confidence level as the target action type.
[0061] Furthermore, as one implementation of the above embodiment, the action type determination module 203 further determines whether the highest confidence level is greater than the confidence level threshold. If so, the predicted action type with the highest confidence level is determined to be the target action type; otherwise, the first frame sequence is determined to have no action recognition result.
[0062] In one embodiment, the action type determination module 203 determines the target action type based on the action type combination sequence. Specifically, the action type determination module 203 includes an action type combination unit, which is used to combine a first action type sequence and a second action type sequence to obtain an action type combination sequence. The second action type sequence is obtained based on a second frame sequence, and the second frame sequence includes multiple video frames of a second video segment that is continuous with the first video segment.
[0063] In one implementation, the action type combination unit includes a first action type part and a second action type part. During the combination process, the first action type sequence output by the model inference module 202 updates the second action type part to form an updated second action type part, and the second action type part before the update updates the first action type part, thereby obtaining an updated action type combination unit.
[0064] In one embodiment, the testing module 204 combines the first Gaussian regression sequence and the second Gaussian regression sequence output by the model inference module 202 to obtain a Gaussian regression combination sequence, and performs a Gaussian distribution test based on the Gaussian regression combination sequence.
[0065] The specific verification module 204 includes a Gaussian regression combination unit for combining the first Gaussian regression sequence and the second Gaussian regression sequence to obtain a Gaussian regression combination sequence. The Gaussian regression combination unit includes a first Gaussian regression part and a second Gaussian regression part. During the combination process, the first Gaussian regression sequence output by the model inference module 202 updates the second Gaussian regression part to obtain the updated second Gaussian regression part. The second Gaussian regression part before the update updates the first Gaussian regression part. After the two Gaussian regression parts are updated, the Gaussian regression combination sequence corresponding to this output can be obtained.
[0066] In one embodiment, the model inference module 202 obtains the action recognition model as follows: First, a Gaussian model-based action recognition model is designed; second, action video samples are collected and labeled; finally, the designed neural network model is used to train the labeled video samples, and the trained model is stored on disk for online inference.
[0067] In the initial model design process, the neural network model consists of three parts: a feature extraction backbone, a single-frame classification head, and a multi-channel Gaussian regression head.
[0068] The feature extraction backbone is used to extract the depth spatiotemporal features of continuous video frames. Commonly used action recognition network backbones can be used in the action recognition model based on Gaussian modeling in this invention, such as 3D convolutional neural network, dual-stream network and VideoSwin Transformer. For scenarios with high real-time requirements, lightweight backbones such as R(2+1)D and TSM can also be selected. In this embodiment, ResNet50-3D is selected as the backbone extraction network.
[0069] The single-frame classification head predicts the action type for each frame based on the deep features extracted from the backbone network, and outputs the prediction result for each frame. In this embodiment, the input initial model has a fixed video frame count of L=64, and the action types include three categories: background, capture, and release, i.e., C=3. Therefore, the output dimension of the single-frame classification head is... The recognition results are used, and the multi-class cross-entropy loss function is used as the loss function for this branch during the training phase. The single-frame classification head branch plays a strong supervisory role in the prediction of each frame, which can increase the generalization ability of the model and improve the convergence speed of the multi-channel Gaussian regression head.
[0070] The multi-channel Gaussian regression head predicts single-frame Gaussian regression values based on the depth features extracted from the backbone. The Gaussian regression values range from [0, 1]. The dimension of the output of this branch is... That is, each type of channel outputs an L-dimensional Gaussian regression vector, and the sigmoid cross-entropy loss function is used as the loss function for that branch during the training phase.
[0071] Video sample annotation includes single-frame action type annotation and single-frame Gaussian distribution annotation.
[0072] First, a series of video samples containing three action categories—background, capture, and release—were collected, and the videos were segmented into frames to obtain a frame sequence image. Secondly, the keyframes of the action are labeled with the tag values 0, 1, ..., C-1, where C represents the total number of action types. In this embodiment, 0 is the background action tag value, 1 is the release action tag value, and 2 is the capture action tag value.
[0073] Based on the keyframe information, generate the action class type and Gaussian regression label value of the associated frames. That is, assuming the keyframe index is μ, the corresponding action type is c, and the action coverage length is 2t+1, the single-frame action category label value associated with keyframe μ can be represented as: ; The single-frame Gaussian distribution annotation value associated with the keyframe μ can be expressed as: ; in, .
[0074] For the labeled frame sequence sample set, the training set and test set are divided according to a certain ratio. In this embodiment, the ratio of the training set to the test set is 9:1, with a total of 500,000 samples. During the training process, First, the frame sequence is uniformly scaled to a size of 124 x 124; Secondly, the frame sequence images are randomly cropped at a size of 112 x 112; Finally, a random selection of consecutive frame segments of length L is fed into the action recognition model based on Gaussian modeling. The model is trained using stochastic gradient descent. In this embodiment, the total number of training rounds is set to 2000. After each round of training, the test set is validated, the mean square error of the multi-channel Gaussian regression values is calculated, and the model with the smallest mean square error of the test set in all rounds is saved to disk for online inference.
[0075] For specific limitations regarding the motion recognition device, please refer to the limitations of the motion recognition method above, which will not be repeated here. Each module in the aforementioned motion recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0076] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0077] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0078] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. An action recognition method, characterized in that, include: Obtain a first frame sequence, the first frame sequence including L frames of the first video segment to be identified; The first frame sequence is input into the action recognition model to obtain the first action type sequence of the video frame and the first Gaussian regression sequence predicted for each action type, wherein the first action type sequence includes the action type predicted for each video frame. The target action type is determined based on the frequency of action types in the first action type sequence; A Gaussian distribution test is performed on the first Gaussian regression sequence corresponding to the target action type to obtain a test value. If the test value is greater than the test threshold, the target action type is determined to be the action recognition result of the first video segment. The method further includes, after obtaining the first action type sequence of the video frame and the first Gaussian regression sequence predicted for each action type, obtaining an action type combination sequence based on the first action type sequence and the second action type sequence, wherein the second action type sequence is obtained based on a second frame sequence, and the second frame sequence includes multiple video frames of a second video segment that is continuous with the first video segment; wherein determining the target action type based on the frequency of action types in the first action type sequence includes determining the target action type based on the frequency of action types in the action type combination sequence. After obtaining the first action type sequence of the video frame and the first Gaussian regression sequence predicted for each action type, the method further includes: obtaining a Gaussian regression combination sequence based on the first Gaussian regression sequence and the second Gaussian regression sequence corresponding to each action type, wherein the second Gaussian regression sequence is obtained based on the second frame sequence; wherein performing a Gaussian distribution test based on the first Gaussian regression sequence corresponding to the target action type includes: performing a Gaussian distribution test based on the Gaussian regression combination sequence corresponding to the target action type to obtain a test value.
2. The action recognition method according to claim 1, characterized in that, The action type combination sequence includes a first action type part and a second action type part; Based on the first action type sequence and the second action type sequence, an action type combination sequence is obtained, including: The second action type portion is updated according to the first action type sequence to obtain the updated second action type portion; The first action type part is updated based on the second action type part before the update to obtain the updated first action type part.
3. The action recognition method according to claim 1, characterized in that, The Gaussian regression combination sequence includes a first Gaussian regression part and a second Gaussian regression part; The step of obtaining a Gaussian regression combination sequence based on the first Gaussian regression sequence and the second Gaussian regression sequence corresponding to each action type includes: The second Gaussian regression part is updated based on the first Gaussian regression sequence corresponding to the action type, resulting in the updated second Gaussian regression part. The first Gaussian regression part is updated based on the second Gaussian regression part before the update to obtain the updated Gaussian regression combination sequence.
4. The action recognition method according to claim 1, characterized in that, Determining the target action type based on the frequency of action types in the first action type sequence includes: The confidence level of each action type is obtained based on its frequency in the first action type sequence. The action type with the highest confidence level is identified as the target action type.
5. The action recognition method according to claim 4, characterized in that, Before determining the action type with the highest confidence level as the target action type, the method further includes: Determine whether the highest confidence level is greater than the confidence threshold. If so, determine the predicted action type with the highest confidence level as the target action type.
6. The action recognition method according to any one of claims 1-5, characterized in that, The action recognition module is trained according to the following steps: Obtain a frame sequence sample set, which includes L training video frames, wherein the training video frames include action type label values and Gaussian distribution annotation values; Input the frame sequence sample set into the initial model to obtain the predicted action type sequence for each training video frame and the predicted Gaussian regression sequence corresponding to each action type; The initial model is iteratively trained based on the action recognition loss to obtain an action recognition model. The action recognition loss includes action type loss and Gaussian regression loss. The action type loss is obtained based on the action type label value and the predicted action type sequence, and the Gaussian regression loss is obtained based on the Gaussian distribution label value and the predicted Gaussian regression sequence.
7. The action recognition method according to claim 6, characterized in that, The Gaussian distribution label value is determined according to the following mathematical expression: ； in, For the first Gaussian distribution annotation values of training video frames. To train the keyframe index in the video frames, For the starting frame index, To terminate the frame index, .
8. A motion recognition device, characterized in that, The device includes: The video frame acquisition module is used to acquire a first frame sequence, the first frame sequence including L video frames of the first video segment to be identified; The model inference module is used to obtain a first action type sequence of the video frame and a first Gaussian regression sequence predicted for each action type after the action recognition model is input into the first frame sequence, wherein the first action type sequence includes the action type predicted for each video frame. An action type determination module is used to determine a target action type based on the frequency of action types in the first action type sequence. After obtaining the first action type sequence of the video frame and the first Gaussian regression sequence predicted for each action type, the module further includes: obtaining an action type combination sequence based on the first action type sequence and the second action type sequence, wherein the second action type sequence is obtained based on a second frame sequence, and the second frame sequence includes multiple video frames of a second video segment that is continuous with the first video segment; wherein determining the target action type based on the frequency of action types in the first action type sequence includes: determining the target action type based on the frequency of action types in the action type combination sequence. After obtaining the first action type sequence of the video frame and the first Gaussian regression sequence predicted for each action type, the method further includes: obtaining a Gaussian regression combination sequence based on the first Gaussian regression sequence and the second Gaussian regression sequence corresponding to each action type, wherein the second Gaussian regression sequence is obtained based on the second frame sequence; wherein, performing a Gaussian distribution test based on the first Gaussian regression sequence corresponding to the target action type includes: performing a Gaussian distribution test based on the Gaussian regression combination sequence corresponding to the target action type to obtain a test value. The verification module is used to perform Gaussian distribution verification based on the first Gaussian regression sequence corresponding to the target action type to obtain a verification value, and if the verification value is greater than the verification threshold, determine that the target action type is the action recognition result of the first video segment.