An action recognition method, apparatus, electronic device, and storage medium

Abstract

This application provides an action recognition method, apparatus, electronic device, and storage medium to improve the accuracy of action recognition. The method includes: acquiring N first videos, each containing a dynamic object, where N is a positive integer; performing occlusion processing on the N first videos to obtain N sets of occluded videos; wherein the occlusion processing includes occlusion of the skeleton portion of the dynamic object and / or occlusion of time blocks in the first videos; determining an action recognition model based on the N sets of occluded videos; wherein the input to the action recognition model includes a second video, and the output of the action recognition model indicates the action type of the dynamic object included in the second video.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an action recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] Action recognition plays a very important role in the field of object behavior recognition. For example, recognizing a dynamic object falling down makes it easier to take relevant measures in a timely manner. Dynamic objects can include dynamic human bodies, dynamic mammals, etc.

[0003] In related technologies, geometric transformations such as rotation, shearing, and random perturbation of joints are mainly applied to video images containing human movements, followed by comparative learning to obtain motion features; or self-supervised learning is used to apply geometric transformations and temporal perturbations to video images containing human movements to obtain motion features. Neither of these methods considers the possibility of occlusion in human movements, meaning the learned motion features are not robust enough, resulting in low accuracy in recognizing human movements and making them unsuitable for scenarios with occlusions. Summary of the Invention

[0004] This application provides an action recognition method, device, electronic device, and storage medium that can improve the accuracy of action recognition.

[0005] In a first aspect, embodiments of this application provide an action recognition method, comprising: acquiring N first videos, each first video including a dynamic object, where N is a positive integer; performing occlusion processing on the N first videos to obtain N sets of occluded videos; wherein the occlusion processing includes occlusion of the skeleton portion of the dynamic object and / or occlusion of time blocks of the first videos; determining an action recognition model based on the N sets of occluded videos; wherein the input of the action recognition model includes a second video, and the output of the action recognition model is used to indicate the action type of the dynamic object included in the second video.

[0006] By training an action recognition model with occlusion, action recognition can be achieved even under occlusion conditions, which can improve the model's generalization performance and the accuracy of action recognition.

[0007] In one optional implementation, determining the action recognition model based on the N sets of occluded videos includes: performing self-supervised learning training using the N sets of occluded videos to obtain a target encoder, the target encoder being able to extract skeleton features from the N sets of occluded videos; and determining the action recognition model based on the target encoder. Occlusion-based self-supervised learning training can enhance the learning of occlusion-robust features.

[0008] In one optional implementation, the Nth group of occluded videos includes N groups of first-type occluded videos and N groups of second-type occluded videos. The occlusion process on the N first videos to obtain the N groups of occluded videos includes: randomly occluding the i-th first video among the N first videos to obtain the i-th group of first-type occluded videos in the N groups of first-type occluded videos; the random occlusion includes occlusion of the skeleton of the dynamic object and / or occlusion of time blocks in the first videos; occluding the skeleton of the dynamic object in the i-th first video among the N first videos to obtain the i-th group of second-type occluded videos in the N groups of second-type occluded videos; where i is a positive integer from 1 to N. By combining skeleton occlusion and time block occlusion, occlusion-based contrastive learning can be achieved, enabling the learning of occlusion-robust features.

[0009] In one optional implementation, the N sets of first-type occlusion videos include N first-occlusion videos and N second-occlusion videos, where the i-th first video corresponds to the i-th first-occlusion video among the N first-occlusion videos and the i-th second-occlusion video among the N second-occlusion videos. The step of using the N sets of occlusion videos for self-supervised learning training to obtain the target encoder includes:

[0010] Based on the N groups of first-class occluded videos, the value of the first loss function corresponding to the occlusion contrast learning model is determined. The occlusion contrast learning model includes a first encoder and a second encoder. The first encoder is used to extract skeleton features from the N first occluded videos, and the second encoder is used to extract skeleton features from the N second occluded videos.

[0011] Based on the N groups of second-class occlusion videos, the value of the second loss function corresponding to the occlusion type classification model is determined; wherein, the occlusion type classification model includes the first encoder and the feature embedding layer, the input of the occlusion type classification model includes the i-th group of second-class occlusion videos, and the output of the occlusion type classification model is used to indicate the occlusion skeleton part corresponding to each occlusion video in the i-th group of second-class occlusion videos;

[0012] Based on the values ​​of the first loss function and the second loss function, the first encoder is updated and trained to obtain the target encoder.

[0013] Through this design, while achieving occlusion-based contrastive learning and learning occlusion-robust features, feature learning for occlusion classification is introduced to assist, enhancing the network's perception of the semantics of skeleton parts and improving the network's ability to model space.

[0014] In one optional implementation, determining the action recognition model based on the target encoder includes: updating the model parameters of the target encoder using model parameters of a preset encoder to obtain an updated target encoder; wherein the preset encoder has the same model structure as the target encoder, and the performance of the preset encoder is greater than a preset performance threshold; determining the action recognition model based on the updated target encoder, the action recognition model including the updated target encoder, a pooling layer, and a fully connected layer. This design optimizes the model parameters of the target encoder, which helps improve the performance of the action recognition model.

[0015] In one optional implementation, the second type of occlusion video includes multiple occlusion videos, and the different occlusion videos correspond to different occlusion skeleton parts.

[0016] In one alternative implementation, the dynamic object includes a dynamic human body, and the skeletal parts include one or more of the following: torso, left arm, right arm, left leg, right leg; or, the dynamic object includes a dynamic mammal, and the skeletal parts include one or more of the following: torso, left forelimb, right forelimb, left hindlimb, right hindlimb.

[0017] Secondly, embodiments of this application provide an action recognition device, comprising:

[0018] The acquisition module is used to acquire N first videos, where each first video includes dynamic objects and N is a positive integer;

[0019] The processing module is used to perform occlusion processing on the N first videos to obtain N sets of occluded videos; wherein, the occlusion processing includes occlusion of the skeleton part of the dynamic object and / or occlusion of the time block of the first video;

[0020] The processing module is further configured to determine an action recognition model based on the N sets of occluded videos; wherein the input of the action recognition model includes the second video, and the output of the action recognition model is used to indicate the action type of the dynamic object included in the second video.

[0021] In an optional implementation, the processing module is further configured to: perform self-supervised learning training using the N sets of occluded videos to obtain a target encoder, the target encoder being able to extract skeleton features from the N sets of occluded videos; and determine the action recognition model based on the target encoder.

[0022] In one optional implementation, the Nth group of occluded videos includes N groups of first-type occluded videos and N groups of second-type occluded videos; the processing module is further configured to: randomly occlude the i-th first video among the N first videos to obtain the i-th group of first-type occluded videos among the N groups of first-type occluded videos, wherein the random occlusion includes occlusion of the skeleton part of the dynamic object and / or occlusion of the time block of the first video; occlude the skeleton part of the dynamic object in the i-th first video among the N first videos to obtain the i-th group of second-type occluded videos among the N groups of second-type occluded videos; wherein i is a positive integer from 1 to N.

[0023] In one optional implementation, the N sets of first-type occlusion videos include N first-occlusion videos and N second-occlusion videos, where the i-th first video corresponds to the i-th first-occlusion video among the N first-occlusion videos and the i-th second-occlusion video among the N second-occlusion videos. When the processing module uses the N sets of occlusion videos to perform self-supervised learning training to obtain the target encoder, it is specifically used for:

[0024] Based on the N groups of first-class occluded videos, the value of the first loss function corresponding to the occlusion contrast learning model is determined. The occlusion contrast learning model includes a first encoder and a second encoder. The first encoder is used to extract skeleton features from the N first occluded videos, and the second encoder is used to extract skeleton features from the N second occluded videos.

[0025] Based on the N groups of second-class occlusion videos, the value of the second loss function corresponding to the occlusion type classification model is determined; wherein, the occlusion type classification model includes the first encoder and the feature embedding layer, the input of the occlusion type classification model includes the i-th group of second-class occlusion videos, and the output of the occlusion type classification model is used to indicate the occlusion skeleton part corresponding to each occlusion video in the i-th group of second-class occlusion videos;

[0026] Based on the values ​​of the first loss function and the second loss function, the first encoder is updated and trained to obtain the target encoder.

[0027] In one optional implementation, when the processing model determines the action recognition model based on the target encoder, it is specifically used to: update the model parameters of the target encoder using the model parameters of a preset encoder to obtain an updated target encoder; wherein the preset encoder has the same model structure as the target encoder, and the performance of the preset encoder is greater than a preset performance threshold; and determine the action recognition model based on the updated target encoder, wherein the action recognition model includes the updated target encoder, a pooling layer, and a fully connected layer.

[0028] In one optional implementation, the second type of occlusion video includes multiple occlusion videos, and the different occlusion videos correspond to different occlusion skeleton parts.

[0029] In one alternative implementation, the dynamic object includes a dynamic human body, and the skeletal parts include one or more of the following: torso, left arm, right arm, left leg, right leg; or, the dynamic object includes a dynamic mammal, and the skeletal parts include one or more of the following: torso, left forelimb, right forelimb, left hindlimb, right hindlimb.

[0030] Thirdly, embodiments of this application provide an electronic device, which includes at least a processor and a memory, wherein the processor executes the steps included in any of the optional embodiments of the first aspect when executing a computer program stored in the memory.

[0031] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the steps included in any of the embodiments of the first aspect. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application.

[0033] Figure 1 A flowchart illustrating an action recognition method provided in an embodiment of this application;

[0034] Figure 2 A schematic diagram of a human skeleton provided for an embodiment of this application;

[0035] Figure 3 A schematic diagram of random occlusion provided for an embodiment of this application;

[0036] Figure 4 A schematic diagram illustrating skeletal part occlusion as provided in an embodiment of this application;

[0037] Figure 5 A schematic diagram of an occlusion-based self-supervised learning framework provided for an embodiment of this application;

[0038] Figure 6 A schematic diagram illustrating an application framework for an action recognition model provided in an embodiment of this application;

[0039] Figure 7 This is a schematic diagram of the structure of an action recognition device provided in an embodiment of this application;

[0040] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0042] The terms "first" and "second" in the embodiments of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0043] In this application embodiment, "at least one" can mean at least two, such as two, three or more, and this application embodiment does not impose any restrictions.

[0044] Furthermore, in the embodiments of this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0045] Action recognition plays a crucial role in object behavior recognition, such as recognizing a dynamic object falling to facilitate timely intervention. Dynamic objects can include dynamic human bodies, dynamic mammals, etc. Related technologies primarily employ geometric transformations such as rotation, shearing, and random joint perturbation on video images containing human movements, followed by comparative learning to obtain action features; or they utilize self-supervised learning to apply geometric transformations and temporal perturbations to video images containing human movements to obtain action features. However, neither of these methods considers the possibility of occlusion during human movement, meaning the learned action features are not robust enough, resulting in low accuracy in recognizing human movements and unsuitability for scenarios with occlusions.

[0046] Based on this, embodiments of this application provide an action recognition method, apparatus, electronic device, and storage medium, which can improve the accuracy of action recognition.

[0047] To better understand, the technical solutions of this application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solutions of this application, rather than limitations thereof. Unless otherwise specified, the embodiments and technical features in the embodiments can be combined with each other.

[0048] Example 1

[0049] See Figure 1 This illustrates an action recognition method, which includes the following steps:

[0050] S101, Obtain N first videos, where each first video includes dynamic objects in motion, and N is a positive integer.

[0051] Optionally, the first video can be video data or video images extracted from a relevant training set; or it can be acquired by a camera device, such as a webcam or camera.

[0052] S102, perform occlusion processing on the N first videos to obtain N sets of occluded videos.

[0053] Specifically, the occlusion process may include spatial occlusion of the skeletal portion of a dynamic object and / or temporal occlusion of a time block of the first video. Optionally, see... Figure 2 This is a schematic diagram of a human skeleton. The skeletal parts that are obscured may include one or more of the following: torso, left arm, right arm, left leg, and right leg. It is understood that the embodiments of this application... Figure 2 The image shown is a human body in the video, a mirror image of a human body in a real scene. Time blocks can represent a portion of a time segment in the first video, such as a time block comprising one or more video frames, consecutive seconds or minutes, or non-consecutive seconds or minutes, etc.

[0054] Specifically, the i-th first video among the N first videos corresponds to the i-th occluded video in the N groups of occluded videos, where i is a positive integer from 1 to N. The N-th group of occluded videos includes N groups of first-type occluded videos and N groups of second-type occluded videos, that is, the i-th group of occluded videos includes the i-th group of first-type occluded videos in the N groups of first-type occluded videos and the i-th group of second-type occluded videos in the N groups of second-type occluded videos.

[0055] The following section, using A1 and A2 as examples, describes in detail how the occlusion process of the i-th first video is performed to obtain the i-th group of first-type occluded videos and the i-th group of second-type occluded videos.

[0056] A1, by randomly occluding the i-th first video, the i-th group of first-type occluded videos can be obtained. The random occlusion includes occlusion of the skeletal part of the dynamic object and / or occlusion of time blocks of the first video. For example, taking a dynamic human body as an example, see [link to relevant documentation]. Figure 3 This is a schematic diagram of random occlusion. Figure 3 The i-th first video is denoted as X, and it comprises 9 time blocks, denoted as t1 to t9. After two random occlusions, the i-th group of first-type occluded videos can be obtained, which includes two first-type occluded videos, denoted as the first occluded video X. ot1 Second occlusion video X ot2 .in, Figure 3 The illustrated random occlusion includes occlusion of skeletal parts and time blocks. X ot1 (X ot2 In the diagram, a "black rectangle" represents an occlusion of a time block within a given time block. X ot1 (X ot2 If the skeleton of the first video X is missing in the same time block, it indicates that there is occlusion of the skeleton. For example, if X is missing at time t3... ot1 Compared to X, which has a missing right arm, this indicates that the right arm is obstructing the view.

[0057] Based on the above, it can be understood that the N groups of first-type occlusion videos include N first-occlusion videos and N second-occlusion videos, and the i-th first video corresponds to the i-th first-occlusion video among the N first-occlusion videos and the i-th second-occlusion video among the N second-occlusion videos.

[0058] A2, by occluding the skeleton part of the dynamic object in the i-th first video, the i-th group of second-type occluded videos can be obtained.

[0059] Optionally, the second type of occlusion video may include multiple occlusion videos, where different occlusion videos correspond to different occlusion skeleton parts. For example, taking a dynamic human body as an example, see [link to relevant documentation]. Figure 4 This is a schematic diagram of skeletal occlusion. Figure 4 The first video, denoted as X, illustrates the processing of five types of skeletal occlusion. The resulting second-class video group consists of five second-class occlusion videos. For example, occluding the right arm in each time block yields X. oc1 X is obtained by occluding the left arm in each time block. oc2 Obscuring the left leg in each time block yields X. oc3Obscuring the right leg in each time block yields X. oc4 And X is obtained by occluding the torso in each time block. oc5 .

[0060] Furthermore, it is understood that the random occlusion described in A1 and the occlusion of the skeleton described in A2 can be processed in parallel or sequentially. For example, random occlusion can be performed first to obtain the first type of occluded video, and then the occlusion of the skeleton can be performed to obtain the second type of occluded video. This application does not impose any limitations on this.

[0061] S103, determine the action recognition model based on the N sets of occluded videos.

[0062] The action recognition model is used to identify the action type of a dynamic object, thereby determining the behavior of the dynamic object. Alternatively, it can be described as follows: the input to the action recognition model includes a second video, and the output of the action recognition model is used to indicate the action type of the dynamic object included in the second video. It is understood that when the dynamic object described in this embodiment is a dynamic human body, the dynamic object can be a user of a related smart device such as a mobile phone, a pedestrian, or related staff, etc.

[0063] Specifically, the scheme for determining the action recognition model can be implemented with reference to B1 and B2 below.

[0064] B1. Self-supervised learning training is performed using the N sets of occluded videos to obtain the target encoder.

[0065] The trained target encoder can be used to extract skeleton features from the N sets of occluded videos.

[0066] First, this self-supervised learning method utilizes an occlusion contrastive learning model, which includes a first encoder and a second encoder. For example, see... Figure 5 This illustrates a self-supervised learning framework based on occlusion. The occlusion contrastive learning model used adopts the mocov3 framework, which includes a query branch and a key branch. The first encoder is the query encoder on the query branch. Figure 5 The diagram shows the first encoder (f) q The second encoder is the key encoder on the key branch. Figure 5 The diagram shows the second encoder (f) kIn addition, the query branch is connected to a global average pooling (GAP) layer, a projected multi-layer perceptron (ProMLP) for mapping, and a predicted multi-layer perceptron (PreMLP) for prediction after the first encoder. The key branch is connected to a global average pooling (GAP) layer and a ProMLP multi-layer perceptron for mapping after the second encoder.

[0067] The neural network structures of the first encoder and the second encoder can be the same, such as using a spatial-temporal graph convolutional network (ST-GCN), a long-short term memory (LSTM) neural network, or a deep convolutional neural network (CNN) as the encoder. This application does not limit this.

[0068] Based on the structure of the occlusion contrastive learning model described above, the N sets of first-class occluded videos can be used as training data for the occlusion contrastive learning model, thereby enabling the updating and training of the first encoder and the second encoder. Specifically, the N first-occluded videos from the N sets of first-class occluded videos can be input to the query branch, and the N second-occluded videos from the N sets of first-class occluded videos can be input to the key branch. It can be understood that the first encoder is used to extract skeleton features from the N first-occluded videos, and the second encoder is used to extract skeleton features from the N second-occluded videos. Figure 5 As an example, it illustrates inputting the i-th first occluded video corresponding to the i-th first video into the query branch, and inputting the i-th second occluded video corresponding to the i-th first video into the key branch.

[0069] Specifically, based on N sets of first-class occlusion videos, the value of the first loss function corresponding to the occlusion contrastive learning model can be determined. As an example, the first loss function can be the InfoNCE loss function. The contrastive learning process can be expressed as follows: (1) to (4):

[0070] q = PreMLP(ProMLP) q (GAP(f q (X ot1 ))) (1)

[0071] k = ProMLPk (GAP(f k (X ot2 (2)

[0072]

[0073] GAP is a global average pooling operation that pools data along the time and skeleton component (e.g., joint) dimensions. (ProMLP) q PreMLP is a multilayer perceptron used for mapping in the query branch, while ProMLP is a multilayer perceptron used for prediction in the query branch. k This is a multilayer perceptron used for mapping in the key branch. `q` is the output of the query branch, and `k` is the output of the key branch. + k represents the positive samples of q corresponding to the same video (e.g., the i-th first video). - It is the set of negative samples of q. τ is an adjustable temperature hyperparameter. The first encoder (f) of the query branch q The model parameters are updated using gradient-based backpropagation, with the second encoder (f) in the key branch. k The model parameters are updated based on momentum updates. Let p be the model parameter of the query branch. q The model parameter for the key branch is p. k The momentum update of the key branch model parameters can be expressed as the following formula (4):

[0074] p k =m×p k +(1-m)×p q (4)

[0075] Where m∈[0,1], m is a momentum coefficient used to ensure smooth parameter updates, which can be pre-configured or dynamically adjusted according to the actual training situation.

[0076] Secondly, this self-supervised learning uses an occlusion type classification model, which includes the first encoder from the aforementioned occlusion contrastive learning model, and a feature embedding layer. The input to the occlusion type classification model includes the i-th group of second-class occluded videos, and the output of the occlusion type classification model is used to indicate the occlusion skeleton part corresponding to each occluded video in the i-th group of second-class occluded videos.

[0077] It should be noted that the first encoder in this occlusion type classification model can be an untrained first encoder, meaning that the aforementioned contrastive learning and occlusion type classification are trained simultaneously; or it can be a first encoder that has already undergone one or more rounds of training through contrastive learning, meaning that the occlusion type classification model is introduced during the contrastive learning training, with contrastive learning occurring some time before occlusion type classification. This application's embodiments do not limit this. As an example, Figure 5 The diagram illustrates that the occlusion type classification model includes a first encoder (f) q ), feature embedding layers, and linear classifiers for classification, such as GAP and fully connected (FC) layers.

[0078] Based on the structure of the above occlusion type classification model, the value of the second loss function corresponding to the occlusion type classification model can be determined according to the N groups of second-class occlusion videos. Taking the i-th group of second-class occlusion videos described in A2 as an example, the five second-class occlusion videos correspond to five occlusion part branches. These five second-class occlusion videos serve as training data for the occlusion type classification model, corresponding to or sharing the same model structure and the same model parameters. The second loss function can be the cross-entropy loss function. Taking the branch of occlusion of the right arm in the i-th group of second-class occlusion videos as an example, the learning process of occlusion type classification can be expressed as the following formulas (5) to (7):

[0079]

[0080]

[0081] Here, embedding represents the feature embedding layer, and the optional structure is a single-layer ST-GCN. FC occu It is a fully connected layer used for occlusion classification. oc1 This corresponds to the output of the branch that is blocked on the right arm. y oc1 This refers to the label corresponding to the obscured right arm branch. N represents the batch size, i.e., the aforementioned N first videos. This is the loss function for occlusion of the right arm branch. The occlusion of the other four parts follows the same logic, and the occlusion of each of the N first videos follows the same logic. Finally, the second loss function of the occlusion type classification model can be expressed as the following formula (7):

[0082]

[0083] Furthermore, the first encoder can be updated and trained based on the values ​​of the first loss function and the second loss function to obtain the target encoder.

[0084] Specifically, the first loss function and the second loss function can be combined to obtain the total loss function for self-supervised learning training, denoted as the following formula (8):

[0085]

[0086] Here, λ is a hyperparameter used to control the second loss function. The weights for self-supervised learning. The total loss function for self-supervised learning training is equal to the sum of the values ​​of the first loss function and the second loss function.

[0087] Then, based on the value of the total loss function of the self-supervised learning training, the model parameters of the first encoder are updated to obtain the target encoder.

[0088] In the self-supervised learning process provided in this application embodiment, occlusion-based contrastive learning is achieved by combining skeleton parts and temporal block occlusion, enabling the learning of occlusion-robust features. Furthermore, feature learning for occlusion classification is introduced to assist in enhancing the network's semantic perception of skeleton parts and improving the network's spatial modeling ability.

[0089] B2, Determine the action recognition model based on the target encoder.

[0090] In one optional implementation, a linear classifier, such as one comprising a GAP layer and a fully connected (FC) layer, can be connected after the target encoder to form an action recognition model. This allows the action recognition model to output the corresponding action classification, or dynamic object behavior, when a video containing dynamic objects is input into the action recognition model.

[0091] In another alternative implementation, supervised learning can be used to fine-tune the target encoder. For example, the model parameters of the target encoder can be updated using the model parameters of a preset encoder to obtain an updated target encoder; wherein the preset encoder has the same model structure as the target encoder, and the performance of the preset encoder is greater than a preset performance threshold.

[0092] Then, based on the updated target encoder, the action recognition model is determined. The action recognition model includes the updated target encoder and a linear classifier, such as a GAP layer and an FC layer.

[0093] Optionally, the aforementioned preset encoder can be a query encoder with good performance obtained through the aforementioned self-supervised learning training. Based on this, the weights f of the query encoder with good performance trained using self-supervised learning can be used. qThe target encoder is initialized, and then a linear classifier is connected for action classification. The loss function for classification is the commonly used cross-entropy loss function. The whole process can be expressed as the following formulas (9) to (10):

[0094] Y i =FC(GAP(f q (X i (9)

[0095]

[0096] Where N represents the batch size, Y i y represents the final output of the action recognition model. i The labels corresponding to the input samples, or training data.

[0097] As an example, let's take a dynamic human body as the example. Figure 6 This illustrates an application framework for an action recognition model. Taking the second video as the i-th first video mentioned above as an example, the second video is input into the action recognition model or an updated action recognition model. The output corresponding to the second video can indicate that the action type of the dynamic object included in the second video is falling.

[0098] The action recognition method provided in this application embodiment is based on occluded learning and training, which can achieve action recognition even under occlusion, effectively improving the generalization performance of the model and the accuracy of action recognition.

[0099] Based on the same concept, embodiments of this application provide an action recognition device that can realize the functions corresponding to the aforementioned action recognition method. This action recognition device can be a hardware structure, a software module, or a hardware structure plus a software module. The action recognition device can be implemented using a chip system, which can consist of chips or include chips and other discrete components. Please refer to... Figure 7 This is a schematic diagram of the structure of an action recognition device provided in an embodiment of this application. The action recognition device includes an acquisition module 701 and a processing module 702.

[0100] Specifically:

[0101] The acquisition module 701 is used to acquire N first videos, where each first video includes dynamic objects and N is a positive integer;

[0102] The processing module 702 is used to perform occlusion processing on the N first videos to obtain N sets of occluded videos; wherein, the occlusion processing includes occlusion of the skeleton part of the dynamic object and / or occlusion of the time block of the first video;

[0103] The processing module 702 is further configured to determine an action recognition model based on the N sets of occluded videos; wherein the input of the action recognition model includes a second video, and the output of the action recognition model is used to indicate the action type of the dynamic object included in the second video.

[0104] In an optional implementation, the processing module 702 is further configured to: perform self-supervised learning training using the N sets of occluded videos to obtain a target encoder, the target encoder being able to extract skeleton features from the N sets of occluded videos; and determine the action recognition model based on the target encoder.

[0105] In one optional implementation, the Nth group of occluded videos includes N groups of first-type occluded videos and N groups of second-type occluded videos; the processing module 702 is further configured to: randomly occlude the i-th first video among the N first videos to obtain the i-th group of first-type occluded videos among the N groups of first-type occluded videos, wherein the random occlusion includes occlusion of the skeleton part of the dynamic object and / or occlusion of the time block of the first video; occlude the skeleton part of the dynamic object in the i-th first video among the N first videos to obtain the i-th group of second-type occluded videos among the N groups of second-type occluded videos; wherein i is a positive integer from 1 to N.

[0106] In one optional implementation, the N sets of first-type occlusion videos include N first-occlusion videos and N second-occlusion videos, where the i-th first video corresponds to the i-th first-occlusion video among the N first-occlusion videos and the i-th second-occlusion video among the N second-occlusion videos. When the processing module 702 uses the N sets of occlusion videos to perform self-supervised learning training to obtain the target encoder, it is specifically used for:

[0107] Based on the N groups of first-class occluded videos, the value of the first loss function corresponding to the occlusion contrast learning model is determined. The occlusion contrast learning model includes a first encoder and a second encoder. The first encoder is used to extract skeleton features from the N first occluded videos, and the second encoder is used to extract skeleton features from the N second occluded videos.

[0108] Based on the N groups of second-class occlusion videos, the value of the second loss function corresponding to the occlusion type classification model is determined; wherein, the occlusion type classification model includes the first encoder and the feature embedding layer, the input of the occlusion type classification model includes the i-th group of second-class occlusion videos, and the output of the occlusion type classification model is used to indicate the occlusion skeleton part corresponding to each occlusion video in the i-th group of second-class occlusion videos;

[0109] Based on the values ​​of the first loss function and the second loss function, the first encoder is updated and trained to obtain the target encoder.

[0110] In one optional implementation, when the processing model determines the action recognition model based on the target encoder, it is specifically used to: update the model parameters of the target encoder using the model parameters of a preset encoder to obtain an updated target encoder; wherein the preset encoder has the same model structure as the target encoder, and the performance of the preset encoder is greater than a preset performance threshold; and determine the action recognition model based on the updated target encoder, wherein the action recognition model includes the updated target encoder, a pooling layer, and a fully connected layer.

[0111] In one optional implementation, the second type of occlusion video includes multiple occlusion videos, and the different occlusion videos correspond to different occlusion skeleton parts.

[0112] In one alternative implementation, the dynamic object includes a dynamic human body, and the skeletal parts include one or more of the following: torso, left arm, right arm, left leg, right leg; or, the dynamic object includes a dynamic mammal, and the skeletal parts include one or more of the following: torso, left forelimb, right forelimb, left hindlimb, right hindlimb.

[0113] Based on the same inventive concept, this application also provides an electronic device, please refer to [link to relevant documentation]. Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device includes at least one processor 802 and a memory 801 connected to or coupled to the at least one processor 802. Furthermore, the electronic device may also include a communication interface 803. The electronic device can interact with other devices through the communication interface 803. For example, the communication interface 803 can be a transceiver, circuit, bus, module, pin, or other type of communication interface. When the electronic device is a chip-type device or circuit, the communication interface 803 in the electronic device can also be an input / output circuit, capable of inputting information (or receiving information) and outputting information (or sending information). The processor is an integrated processor, microprocessor, integrated circuit, or logic circuit, and the processor can determine the output information based on the input information.

[0114] The coupling in this application embodiment is an indirect coupling or communication connection between devices, units, or modules, which can be electrical, mechanical, or other forms, used for information interaction between devices, units, or modules. The processor 802 may operate in conjunction with the memory 801 and the communication interface 803. This disclosure does not limit the specific connection medium between the processor 802, the memory 801, and the communication interface 803.

[0115] Optional, see Figure 8 The processor 802, the memory 801, and the communication interface 803 are interconnected via a bus 840. The bus 800 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0116] In this embodiment, memory 801, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 801 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory 801 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. Memory 801 in this embodiment may also be a circuit or any other device capable of implementing storage functions for storing instructions, computer programs, and / or data.

[0117] In this embodiment, the processor 802 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in this embodiment. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the action recognition method disclosed in this embodiment can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0118] By designing and programming the processor 802, the code corresponding to the action recognition method described in the foregoing embodiments can be embedded into the chip, so that the chip can execute the steps of the aforementioned action recognition method when running. How to design and program the processor 802 is a well-known technique to those skilled in the art, and will not be described in detail here.

[0119] Specifically, in this embodiment, the memory 801 stores instructions executable by at least one processor 802. The at least one processor 802 can execute the steps included in the aforementioned action recognition method by calling the instructions or computer program stored in the memory 801. For example, the processor 802 can acquire N first videos through the communication interface 803, where the first videos include dynamic objects, and N is a positive integer. The processor 802 can also perform occlusion processing on the N first videos to obtain N sets of occluded videos; wherein the occlusion processing includes occlusion of the skeleton portion of the dynamic object and / or occlusion of time blocks of the first videos; the processor 802 can also determine an action recognition model based on the N sets of occluded videos; wherein the input of the action recognition model includes a second video, and the output of the action recognition model is used to indicate the action type of the dynamic object included in the second video.

[0120] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the steps of the aforementioned action recognition method.

[0121] In some possible implementations, various aspects of the action recognition method provided in this application may also be implemented in the form of a computer program product, which includes program code that, when the computer program product is run on an electronic device, causes the electronic device to perform the steps in the action recognition method according to the various exemplary embodiments of this application described above.

[0122] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0123] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0124] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0125] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0126] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of the embodiments of this application and their equivalents, then the embodiments of this application are also intended to include these modifications and variations.

Claims

1. An action recognition method, characterized in that, include: Obtain N first videos, where each first video includes dynamic objects, and N is a positive integer; The N first videos are occluded to obtain N sets of occluded videos; wherein, the N sets of occluded videos include N sets of first-type occluded videos and N sets of second-type occluded videos; the occlusion processing corresponding to the N sets of first-type occluded videos includes occlusion of the skeleton part of the dynamic object and / or occlusion of the time block of the first video; the occlusion processing corresponding to the N sets of second-type occluded videos includes occlusion of the skeleton part of the dynamic object. A target encoder is obtained by performing self-supervised learning training using the N sets of occluded videos. The target encoder can be used to extract skeleton features from the N sets of occluded videos. An action recognition model is determined based on the target encoder. The input of the action recognition model includes a second video, and the output of the action recognition model is used to indicate the action type of the dynamic object included in the second video. During the self-supervised learning training process, the N sets of first-type occluded videos are used for the occlusion contrast learning task, and the N sets of second-type occluded videos are used for the occlusion type classification task. The occlusion contrast learning task and the occlusion type classification task share the same encoder and jointly optimize the parameters of the same encoder.

2. The method as described in claim 1, characterized in that, The process of occluding the N first videos yields N sets of occluded videos, including: Randomly occlude the i-th first video among the N first videos to obtain the i-th group of first-type occluded videos among the N groups of first-type occluded videos. The random occlusion includes occlusion of the skeleton part of the dynamic object and / or occlusion of the time block of the first video. Occlude the skeleton of the dynamic object in the i-th first video among the N first videos to obtain the i-th group of second-type occluded videos among the N groups of second-type occluded videos; where i is a positive integer from 1 to N.

3. The method as described in claim 2, characterized in that, The N groups of first-type occlusion videos include N first occlusion videos and N second occlusion videos, where the i-th first video corresponds to the i-th first occlusion video among the N first occlusion videos and the i-th second occlusion video among the N second occlusion videos.

4. The method as described in claim 3, characterized in that, The step of using the N sets of occluded videos for self-supervised learning training to obtain the target encoder includes: Based on the N groups of first-class occluded videos, the value of the first loss function corresponding to the occlusion contrast learning model is determined. The occlusion contrast learning model includes a first encoder and a second encoder. The first encoder is used to extract skeleton features from the N first occluded videos, and the second encoder is used to extract skeleton features from the N second occluded videos. Based on the N groups of second-class occlusion videos, the value of the second loss function corresponding to the occlusion type classification model is determined; wherein, the occlusion type classification model includes the first encoder and the feature embedding layer, the input of the occlusion type classification model includes the i-th group of second-class occlusion videos, and the output of the occlusion type classification model is used to indicate the occlusion skeleton part corresponding to each occlusion video in the i-th group of second-class occlusion videos; Based on the values ​​of the first loss function and the second loss function, the first encoder is updated and trained to obtain the target encoder.

5. The method according to any one of claims 2-4, characterized in that, Determining the action recognition model based on the target encoder includes: The model parameters of the target encoder are updated using the model parameters of the preset encoder to obtain the updated target encoder; wherein the model structure of the preset encoder and the target encoder are the same, and the performance of the preset encoder is greater than a preset performance threshold. Based on the updated target encoder, the action recognition model is determined, and the action recognition model includes the updated target encoder, a pooling layer, and a fully connected layer.

6. The method according to any one of claims 2-4, characterized in that, The second type of occlusion video includes multiple occlusion videos, and the different occlusion videos correspond to different occlusion skeleton parts.

7. The method according to any one of claims 2-4, characterized in that, The dynamic object includes a dynamic human body, and the skeletal parts include one or more of the following: torso, left arm, right arm, left leg, and right leg; or, The dynamic object includes a dynamic mammal, and the skeletal parts include one or more of the following: trunk, left forelimb, right forelimb, left hindlimb, and right hindlimb.

8. A motion recognition device, characterized in that, include: The acquisition module is used to acquire N first videos, where each first video includes dynamic objects and N is a positive integer; The processing module is used to perform occlusion processing on the N first videos to obtain N sets of occluded videos; wherein, the N sets of occluded videos include N sets of first-type occluded videos and N sets of second-type occluded videos; the occlusion processing corresponding to the N sets of first-type occluded videos includes occlusion of the skeleton part of the dynamic object and / or occlusion of the time block of the first video; the occlusion processing corresponding to the N sets of second-type occluded videos includes occlusion of the skeleton part of the dynamic object; The processing module is further configured to perform self-supervised learning training using the N sets of occluded videos to obtain a target encoder, which can be used to extract skeleton features from the N sets of occluded videos; and to determine an action recognition model based on the target encoder; wherein the input of the action recognition model includes a second video, and the output of the action recognition model is used to indicate the action type of the dynamic object included in the second video; during the self-supervised learning training process, the N sets of first-type occluded videos are used for an occlusion contrast learning task, the N sets of second-type occluded videos are used for an occlusion type classification task, and the occlusion contrast learning task and the occlusion type classification task share the same encoder and jointly optimize the parameters of the same encoder.

9. An electronic device, characterized in that, The electronic device includes at least a processor and a memory, wherein the processor is used to execute the steps of any of the action recognition methods described in claims 1-7 when executing a computer program stored in the memory.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the steps of the method according to any one of claims 1-7.

Images