Training method of action recognition model, action recognition method and related device

Abstract

The present application provides a training method for an action recognition model, an action recognition method and related devices, wherein the action recognition model includes a first model and a second model, and the first model includes a third sub-model and a fourth sub-model. The method includes: the first model uses a two-dimensional posture sequence of a character and a three-dimensional posture sequence of a character to train the first model, and the two-dimensional posture sequence of the character and the three-dimensional posture sequence of the character have a consistent connection. The second model uses the two-dimensional posture sequence output by the first model and the two-dimensional posture sequence expanded by the three-dimensional posture output by the first model to train the second model. The training method of the action recognition model provided in the present application can improve the accuracy and recognition efficiency of character action recognition in small sample action recognition scenarios.

Description

Technical Field

[0001] The present application relates to the field of image processing, and more specifically, to a training method for an action recognition model, an action recognition method, and related devices. Background Art

[0002] In many scenarios, it's necessary to identify human motion in videos to determine their types. For example, in public places, human motion recognition is necessary to prevent incidents such as climbing, fighting, and illegal operations that endanger personal and public safety. Similarly, in human-computer interaction systems, human motion recognition is necessary to understand human behavior.

[0003] To achieve human action recognition in videos, existing solutions include action recognition based on 2D pose estimation and action recognition based on 3D pose estimation. 2D pose estimation-based action recognition lacks depth information about the human pose, making this method primarily suitable for scenarios with a large number of action image sequences from target perspectives. 3D pose estimation-based action recognition, on the other hand, incorporates depth information about the human pose, making it primarily suitable for action recognition scenarios with abundant computational resources and less demanding real-time performance.

[0004] Existing action recognition technology suffers from the following major issues: First, it cannot effectively train for small sample sizes or specific viewpoints where sample sizes are missing, resulting in low action recognition accuracy. Second, it cannot effectively apply to action recognition scenarios with limited computing resources and high real-time requirements while ensuring accurate pose estimation and action recognition. Summary of the Invention

[0005] The present application provides a training method for an action recognition model, an action recognition method and related devices. When using this method to recognize human actions in a small sample of human images to be recognized, it has high recognition accuracy and recognition efficiency.

[0006] In a first aspect, a method for training an action recognition model is provided, wherein the action recognition model includes a first model and a second model, and the method includes:

[0007] Obtain M training images and annotation data for the M training images, wherein each of the M training images includes a person, and the annotation data for each training image includes a bounding box containing the person in each training image and annotation information of the person's posture in each training image, where M is an integer greater than 1;

[0008] The M training images are processed by the first model to obtain posture sequences corresponding to the M training images, wherein the posture sequences corresponding to the M training images include a first type of two-dimensional posture sequence corresponding to the M training images and a second type of two-dimensional posture sequence corresponding to the M training images, the first type of two-dimensional posture sequence corresponding to the M training images is obtained based on a two-dimensional heat map of a person included in the M training images, the second type of two-dimensional posture sequence corresponding to the M training images is obtained by performing a two-dimensional projection on the second type of three-dimensional posture sequence corresponding to the M training images, the second type of three-dimensional posture sequence corresponding to the M training images is obtained by rotating the first type of three-dimensional posture sequence corresponding to the M training images by an arbitrary or specific angle, and the first type of three-dimensional posture sequence corresponding to the M training images is obtained based on the three-dimensional heat map of the person included in the M training images;

[0009] Training the second model by using the posture sequences corresponding to the M training images until the second model meets the second preset requirement;

[0010] The first model is trained by the following steps:

[0011] Step 1: Process the person pose annotation information in each training image to obtain the person pose label in each training image, where the pose label includes a two-dimensional pose label, a three-dimensional pose label, and a two-dimensional pose key point confidence heat map label;

[0012] Step 2: Initializing the first model to obtain initial values ​​of model parameters of the first model;

[0013] Step 3: inputting the first batch of training images among the M training images into the first model to obtain a posture sequence corresponding to the first batch of training images, wherein the posture sequence corresponding to the first batch of training images includes a first type of two-dimensional posture sequence corresponding to the first batch of training images, a first type of three-dimensional posture sequence corresponding to the first batch of training images, a common two-dimensional posture sequence corresponding to the first batch of training images, and a common three-dimensional posture sequence corresponding to the first batch of training images, wherein the first type of two-dimensional posture sequence corresponding to the first batch of training images is obtained based on the two-dimensional heat map of the characters included in the first batch of training images, the first type of three-dimensional posture sequence corresponding to the first batch of training images is obtained based on the three-dimensional heat map of the characters included in the first batch of training images, the common two-dimensional posture sequence is a two-dimensional posture sequence determined based on the common posture key points in the two-dimensional posture sequence corresponding to the first batch of training images and the three-dimensional posture sequence corresponding to the first batch of training images, and the common three-dimensional posture sequence is a three-dimensional posture sequence determined based on the common posture key points in the two-dimensional posture sequence corresponding to the first batch of training images and the three-dimensional posture sequence corresponding to the first batch of training images;

[0014] Step 4: Determine the loss value of the first model based on the posture sequence corresponding to the first batch of training images. The loss value of the first model is obtained by weighted processing of the first loss value, the second loss value, the third loss value, and the fourth loss value. The first loss value is determined based on the first type of two-dimensional posture sequence corresponding to the first batch of training images and the two-dimensional posture labels corresponding to the first batch of training images. The second loss value is determined based on the first type of three-dimensional posture sequence corresponding to the first batch of training images and the three-dimensional posture labels corresponding to the first batch of training images. The third loss value is determined based on the common two-dimensional posture sequence corresponding to the first batch of training images and the common three-dimensional posture sequence corresponding to the first batch of training images. The fourth loss value is determined based on the two-dimensional posture key point confidence heat map labels corresponding to the first batch of training images and the two-dimensional posture key point confidence heat map corresponding to the first batch of training images.

[0015] Step 6: Update the model parameters of the first model according to the loss value of the first model;

[0016] Repeat steps 3 to 5 above until the first model meets the first preset requirement.

[0017] Based on the above technical solution, the first model is trained using a two-dimensional pose sequence and a three-dimensional pose sequence of a person, and the two-dimensional pose sequence and the three-dimensional pose sequence of the person have a consistent relationship. Because the third sub-model in the first model takes into account the three-dimensional pose information of the person's pose when predicting the two-dimensional pose of the person, the prediction result of the third sub-model for predicting the two-dimensional pose of the person has higher accuracy and lower prediction delay. The second model can be trained using the two-dimensional pose sequence output by the first model and the two-dimensional pose sequence expanded by the three-dimensional pose output by the first model.

[0018] The training method of the action recognition model provided in the present application can improve the accuracy and efficiency of human action recognition in small-sample action recognition scenarios. Specifically, by using a three-dimensional posture that is consistent with the two-dimensional posture to perform perspective transformation and plane projection in any direction, a two-dimensional posture of the same action under different perspectives is generated, which can effectively improve data diversity and model generalization, thereby improving the accuracy of action recognition in small-sample action recognition scenarios. By using only the third sub-model in the first model to predict the posture of the person, while ensuring that the prediction result of the two-dimensional posture of the person has a high accuracy, it can meet the low posture prediction delay requirement, thereby improving the recognition efficiency and recognition accuracy of the action recognition model for human action recognition.

[0019] In conjunction with the first aspect, in certain implementations of the first aspect, the first model includes a first sub-model and a second sub-model, the second sub-model includes a third sub-model and a fourth sub-model, and processing the M training images using the first model to obtain a pose sequence corresponding to the M training images includes:

[0020] Processing the M training images by the third sub-model to obtain a first type of two-dimensional posture sequence corresponding to the M training images;

[0021] Processing the M training images by the fourth sub-model to obtain a first type of three-dimensional pose sequence corresponding to the M training images;

[0022] The first model is used to rotate the first type of three-dimensional posture sequence corresponding to the M training images at any or specific angle to obtain the second type of two-dimensional posture sequence corresponding to the M training images.

[0023] Based on the above technical solution, rotating the first type of 3D pose sequence at any or specific angles can expand the sample capacity of training images and the richness of the perspectives of the character poses included in the training images. The richness of the perspectives of the character poses included in the training images can be understood as the corresponding poses of the character from different perspectives or specific perspectives.

[0024] In combination with the first aspect, in some implementations of the first aspect, the second model is trained by the following steps:

[0025] Step 7: Initializing the model parameters of the second model to obtain initial values ​​of the model parameters of the second model;

[0026] Step 8: splicing the key point confidences of the two-dimensional pose sequences corresponding to the second batch of training images in the M training images and the first type of two-dimensional pose sequences corresponding to the second batch of training images to obtain a feature map, wherein the key point confidences of the first type of two-dimensional pose sequences corresponding to the second batch of training images are determined by the first model based on the first type of two-dimensional pose sequences corresponding to the second batch of training images, the first type of three-dimensional pose sequences corresponding to the second batch of training images, and the pose key point heat map corresponding to the first type of two-dimensional pose sequences corresponding to the second batch of training images;

[0027] Step 9: Input the feature map into the second model to obtain the loss value of the second model;

[0028] Step 10: updating the model parameters of the second model according to the loss value of the second model;

[0029] Repeat steps 8 to 10 until the second model meets the second preset requirement.

[0030] Based on the above technical solution, the second model can be trained using the two-dimensional posture sequence output by the first model, which can improve the training efficiency of the second model.

[0031] It should be understood that the first batch of training images and the second batch of training images may be the same batch of training images or different batches of training images, and this application does not make any specific limitation thereto.

[0032] In combination with the first aspect, in certain implementations of the first aspect, the two-dimensional pose sequences corresponding to the second batch of training images in the M training images and the key point confidences of the first type of two-dimensional pose sequences corresponding to the second batch of training images are spliced ​​to obtain a feature map, including:

[0033] splicing the key point confidences of the second type of two-dimensional pose sequence corresponding to the second batch of training images and the first type of two-dimensional pose sequence corresponding to the second batch of training images to obtain the feature map; or

[0034] The first type of two-dimensional pose sequence corresponding to the second batch of training images, the second type of two-dimensional pose sequence corresponding to the second batch of training images, and the key point confidence of the first type of two-dimensional pose sequence corresponding to the second batch of training images are spliced ​​to obtain the feature map. In conjunction with the first aspect, in certain implementations of the first aspect, the second model meets the second preset requirement, including:

[0035] The second model meets the second preset requirement when at least one of the following conditions is met:

[0036] The number of training times of the second model is greater than or equal to a second preset number;

[0037] The loss value of the second model is less than or equal to a second preset threshold.

[0038] Optionally, in some implementations, the loss value of the second model is less than or equal to a second preset threshold, which may also be replaced by a change in the loss value of the second model is less than or equal to a third preset threshold.

[0039] In conjunction with the first aspect, in certain implementations of the first aspect, the first model satisfies the first preset requirement, including:

[0040] The first model meets the first preset requirement when at least one of the following conditions is met:

[0041] The number of training times of the first model is greater than or equal to a first preset number;

[0042] The loss value of the first model is less than or equal to a first preset threshold.

[0043] Optionally, in some implementations, the loss value of the first model is less than or equal to a second preset threshold, which may also be replaced by a change in the loss value of the first model is less than or equal to a fourth preset threshold.

[0044] In a second aspect, a motion recognition method is provided, characterized in that the method includes:

[0045] Obtain a sequence of images of people to be identified;

[0046] Processing the image sequence of the person to be identified using the first model to obtain a two-dimensional pose sequence corresponding to the image sequence of the person to be identified and key point confidences of the two-dimensional pose sequence, wherein the two-dimensional pose sequence is obtained based on a two-dimensional heat map of the person included in the image sequence of the person to be identified;

[0047] The two-dimensional posture sequence and the confidence of the key points of the two-dimensional posture sequence are spliced ​​to obtain a feature map, where the confidence of the key points of the two-dimensional posture sequence is determined based on the two-dimensional posture sequence and the posture key point heat map corresponding to the two-dimensional posture sequence;

[0048] Inputting the feature map into the second model to obtain an action recognition result of the image sequence of the person to be recognized;

[0049] The first model and the second model are obtained by training using the training method in the first aspect.

[0050] Based on the above technical solution, the action recognition model provided by the present application has high real-time performance when identifying the actions of people in the image sequence of the people to be identified, and the obtained action recognition results have high accuracy.

[0051] In conjunction with the second aspect, in some implementations of the second aspect, the first model includes a third sub-model,

[0052] The method of using the first model to process the image sequence of the person to be identified to obtain a two-dimensional posture sequence corresponding to the image sequence of the person to be identified and confidences of key points of the two-dimensional posture sequence includes:

[0053] The image sequence of the person to be identified is processed by the third sub-model to obtain a two-dimensional posture sequence corresponding to the image sequence of the person to be identified and confidences of key points of the two-dimensional posture sequence.

[0054] In a third aspect, a training device for an action recognition model is provided, wherein the training device for an action recognition model includes a module for executing the method in the first aspect or any possible implementation of the first aspect.

[0055] In a fourth aspect, a motion recognition device is provided, which includes a module for executing the method in the second aspect or any possible implementation of the second aspect.

[0056] In a fifth aspect, a training device for an action recognition model is provided, wherein the device for the action recognition model includes a storage medium and a central processing unit. The storage medium may be a non-volatile storage medium, in which a computer executable program is stored. The central processing unit is connected to the non-volatile storage medium and executes the computer executable program to implement the method of the first aspect or any possible implementation of the first aspect.

[0057] In a sixth aspect, a motion recognition device is provided, which includes a storage medium and a central processing unit. The storage medium may be a non-volatile storage medium, in which a computer executable program is stored. The central processing unit is connected to the non-volatile storage medium and executes the computer executable program to implement the method in the second aspect or any possible implementation of the second aspect.

[0058] In a seventh aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface is used to communicate with an external device, and the processor is used to execute the method in the first aspect or any possible implementation of the first aspect.

[0059] Optionally, as an implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method in the first aspect or any possible implementation of the first aspect.

[0060] In an eighth aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface is used to communicate with an external device, and the processor is used to execute the method in the second aspect or any possible implementation of the second aspect.

[0061] Optionally, as an implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method in the second aspect or any possible implementation of the second aspect.

[0062] In a ninth aspect, a computer-readable storage medium is provided, wherein the computer-readable medium stores a program code for execution by a device, wherein the program code includes instructions for executing the method in the first aspect or any possible implementation of the first aspect.

[0063] In a tenth aspect, a computer-readable storage medium is provided, wherein the computer-readable medium stores program code for execution by a device, wherein the program code includes instructions for executing the method in the second aspect or any possible implementation of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 1 is a schematic diagram of a system architecture 100 according to an embodiment of the present application.

[0065] Figure 2 2 is a schematic diagram of a system architecture 200 according to an embodiment of the present application.

[0066] Figure 3 It is a schematic flowchart of the training method 100 of the first model in the action recognition model provided in this application.

[0067] Figure 4 It is a schematic diagram of the key points of the character posture provided by this application.

[0068] Figure 5 It is a schematic flowchart of the training method 200 of the second model in the action recognition model provided in this application.

[0069] Figure 6 It is a schematic flow chart of the action recognition method 300 according to an embodiment of the present application.

[0070] Figure 7 It is a schematic block diagram of a training device 7000 for a motion recognition model according to an embodiment of the present application.

[0071] Figure 8 It is a schematic block diagram of a training device 8000 for a motion recognition model according to an embodiment of the present application.

[0072] Figure 9 is a schematic block diagram of the motion recognition device 9000 according to an embodiment of the present application.

[0073] Figure 10 is a schematic block diagram of the motion recognition device 1000 according to an embodiment of the present application. DETAILED DESCRIPTION

[0074] The technical solution in this application will be described below with reference to the accompanying drawings.

[0075] In the embodiments of this application, words such as "exemplary" and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete way.

[0076] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0077] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0078] Below, possible application scenarios of the action recognition method according to the embodiment of the present application are introduced.

[0079] The action recognition method of the embodiment of the present application can be applied in the following scenarios.

[0080] Scenario 1: Pedestrian Behavior Analysis and Early Warning System

[0081] In an autonomous driving system (ADS) or an advanced driving assistant system (ADAS), how to avoid collisions with pedestrians is a very important issue. In order to solve this problem, pedestrians on the road are generally identified and the distance between the pedestrians and the car is calculated to avoid collisions with pedestrians. However, simply calculating the distance between the pedestrians and the car is not a good way to avoid collisions with pedestrians. The motion recognition method of the present application can be applied to a pedestrian behavior analysis and early warning system to identify the motion type of the pedestrian and then obtain the motion type of the pedestrian. Next, the degree of threat to driving can be analyzed based on the motion type of the pedestrian. For example, the walking of a pedestrian poses a smaller threat to driving, while the running or looking at a mobile phone poses a greater threat to driving. By identifying behaviors that pose a greater threat to driving, early warnings can be issued or corresponding treatment measures can be taken.

[0082] Scenario 2: Driver Dangerous Behavior Reminder System

[0083] Many traffic accidents are caused by driver negligence, such as looking at a mobile phone while driving or engaging in other dangerous behaviors while driving. The motion recognition method of this application can be applied to a driver dangerous behavior reminder system to promptly detect the driver's dangerous behavior and issue a warning, thereby reducing the incidence of traffic accidents.

[0084] Scenario 3: Human-computer interaction system

[0085] In a human-computer interaction system, the action recognition method and posture estimation method of the embodiments of the present application can be used to process a player's video, thereby identifying human actions (such as attacking, defending, etc.) and postures (the positions of major human joints). The human-computer interaction system can then perform corresponding operations (such as attacking, defending, etc.) based on the identified actions, and display a character performing the same actions as the player on the screen based on the identified posture, thereby improving the human-computer interaction experience.

[0086] Scenario 4: Dangerous Behavior Identification System in Safe Cities

[0087] The dangerous behavior identification system in a safe city can include city surveillance cameras, a motion recognition module, and a dangerous behavior alarm module. The dangerous behavior identification system uses the video captured by the city surveillance cameras and the motion recognition module (the motion recognition module can specifically use the motion recognition method of the embodiment of the present application to identify human motion) to identify the movements of people in the surveillance video. When people in the surveillance video exhibit dangerous behavior (such as robbery, assault, fighting, etc.), the danger alarm module sends an alarm message to relevant departments.

[0088] It should be understood that, in addition to the above-mentioned scenes 1 to 4, the action recognition method of the embodiment of the present application can also be applied to other scenes that require action recognition of the human body in the video.

[0089] The following combination Figure 1 The system architecture of the embodiment of the present application is introduced in detail.

[0090] Figure 1 Schematic diagram of the system architecture 100 of the embodiment of the present application. Figure 1 As shown, the system architecture 100 includes an execution device 110 , a training device 120 , a database 130 , a client device 140 , a data storage system 150 , and a data collection system 160 .

[0091] In addition, the execution device 110 includes a calculation module 111, an I / O interface 112, a preprocessing module 113, and a preprocessing module 114. The calculation module 111 may include the target model / rule 101, and the preprocessing module 113 and the preprocessing module 114 are optional.

[0092] Data acquisition device 160 is used to collect training data. For the action recognition network training method of the present embodiment, the training data may include M training images and labeled data for the M training images. After collecting the training data, data acquisition device 160 stores the training data in database 130. Training device 120 trains the target model / rule 101 based on the training data maintained in database 130.

[0093] The following describes how the training device 120 obtains the target model / rule 101 based on the training data. The training device 120 performs feature extraction on the input training image to obtain a feature map of the training image, and repeatedly performs feature extraction on the input training image until the function value of the loss function meets the preset requirements (less than or equal to the preset threshold), thereby completing the training of the target model / rule 101.

[0094] It should be understood that the training of the target model / rule 101 can be an unsupervised training.

[0095] The above-mentioned target model / rule 101 can be used to implement the action recognition method of the embodiment of the present application, that is, the image to be recognized (the image to be recognized may be an image that needs to be recognized) is input into the target model / rule 101, and a feature map of the image to be recognized is extracted, and action recognition is performed based on the extracted feature map to determine the recognition result of the action. The target model / rule 101 in the embodiment of the present application can specifically be a neural network. It should be noted that in actual applications, the training data maintained in the database 130 does not necessarily all come from the collection of the data acquisition device 160, and may also be received from other devices. It should also be noted that the training device 120 does not necessarily train the target model / rule 101 based entirely on the training data maintained by the database 130, and may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiment of the present application.

[0096] The target model / rule 101 obtained by training the training device 120 can be applied to different systems or devices, such as Figure 1 The execution device 110 shown in the figure can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR), a vehicle terminal, etc. It can also be a server or a cloud. Figure 1 In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for exchanging data with an external device. A user can input data into the I / O interface 112 via a client device 140. The input data may include an image to be recognized input by the client device in this embodiment. The client device 140 may specifically be a monitoring device.

[0097] Preprocessing module 113 and preprocessing module 114 are used to preprocess input data (such as an image to be recognized) received by I / O interface 112. In the embodiment of the present application, preprocessing module 113 and preprocessing module 114 may be absent or there may be only one preprocessing module. If preprocessing module 113 and preprocessing module 114 are absent, computing module 111 may be used directly to process the input data.

[0098] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 150.

[0099] Finally, the I / O interface 112 presents the processing result (specifically, a high-quality image obtained by action recognition), such as the action recognition result of the image to be recognized obtained by performing action recognition processing on the image to be recognized by the target model / rule 101, to the client device 140, thereby providing it to the user.

[0100] Specifically, the high-quality image obtained by action recognition through the target model / rule 101 in the computing module 111 can be processed (for example, image rendering processing) by the preprocessing module 113 (and optionally the preprocessing module 114), and the processing result can be sent to the I / O interface, and then the I / O interface sends the processing result to the client device 140 for display.

[0101] It should be understood that when the pre-processing module 113 and the pre-processing module 114 do not exist in the above-mentioned system architecture 100, the computing module 111 can also transmit the high-quality image obtained through the motion recognition processing to the I / O interface, and then the I / O interface sends the processing results to the client device 140 for display.

[0102] It is worth noting that the training device 120 can target different goals or different tasks (for example, the training device can be trained on real high-quality images and approximate low-quality images in different scenarios), and generate corresponding target models / rules 101 based on different training data. The corresponding target models / rules 101 can be used to achieve the above goals or complete the above tasks, thereby providing the user with the desired results.

[0103] It is worth noting that Figure 1 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, Figure 1 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110.

[0104] like Figure 1 As shown, the target model / rule 101 obtained through training by the training device 120 can be a neural network model. For example, the neural network model can be a hierarchical co-occurrence network (HCN). For example, the neural network model can be a spatial-temporal graph convolutional neural network (ST-GCN), etc.

[0105] Figure 2 2 is a schematic diagram of a system architecture 200 according to an embodiment of the present application.

[0106] like Figure 2 As shown, the system architecture 200 includes a local device 301, a local device 302, an execution device 210 and a data storage system 250, wherein the local device 301 and the local device 302 are connected to the execution device 210 via a communication network.

[0107] The execution device 210 can be implemented by one or more servers. Optionally, the execution device 210 can be used in conjunction with other computing devices, such as data storage devices, routers, load balancers, and the like. The execution device 210 can be deployed at a single physical site or distributed across multiple physical sites. The execution device 210 can use data in the data storage system 250 or call program code in the data storage system 250 to implement the motion recognition method of the embodiment of the present application.

[0108] Users can operate their respective user devices (e.g., local device 301 and local device 302) to interact with execution device 210. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smartphone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0109] Each user's local device can interact with the execution device 210 through a communication network of any communication mechanism / communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0110] In one implementation, the local device 301 and the local device 302 obtain relevant parameters of the target neural network from the execution device 210, deploy the target neural network on the local device 301 and the local device 302, and use the target neural network to perform action recognition.

[0111] In another implementation, the target neural network can be directly deployed on the execution device 210. The execution device 210 obtains human body images from the local device 301 and the local device 302 (the local device 301 and the local device 302 can upload the human body images to the execution device 210), performs action recognition on the human body images according to the target neural network, and sends the high-quality images obtained by action recognition to the local device 301 and the local device 302.

[0112] The execution device 210 may also be referred to as a cloud device. In this case, the execution device 210 is generally deployed in the cloud.

[0113] Figure 3 It is a schematic flowchart of the training method 100 of the first model in the action recognition model provided in this application.

[0114] like Figure 3 As shown, the method 100 includes steps 1001 to 1009, and steps 1001 to 1009 are described in detail below.

[0115] Step 1001, start.

[0116] The above step 1001 indicates starting the training process of the first model.

[0117] Step 1002: Obtain training data.

[0118] The training data in step 1002 includes M (M is an integer greater than 1) training images and annotation data for the M training images. Each of the M training images includes a person, and the annotation data for each training image includes a bounding box containing the person in each training image and annotation information about the person's posture in each training image.

[0119] In this application, the character posture annotation information is not specifically limited. For example, the character posture annotation information can be standing, falling, or climbing.

[0120] Step 1003: Obtain a posture label of the person included in each training image based on the labeled data of the person included in each training image.

[0121] The person's posture label included in each training image includes the person's two-dimensional posture label, the person's three-dimensional posture label, and a confidence heat map label.

[0122] Taking the i-th training image among M training images as an example, this paper introduces a method for obtaining a person's posture label based on labeled data.

[0123] Specifically, the i-th training image is first intercepted according to the rectangular frame where the target person is located in the i-th training image to generate an image I centered on the target person. Then, the posture annotation information of the person included in the image I is transformed (the posture annotation represented by the original image coordinate system is translated into the posture annotation represented by the image I coordinate system centered on the target person. Specifically, the coordinates of the upper left corner of the image I in the original image are subtracted from the original posture annotation) to generate a two-dimensional posture label of the person included in the image I. 3D pose label and confidence heatmap labels

[0124] Step 1004: Initialize the model parameters of the first model to obtain initial values ​​of the model parameters of the first model.

[0125] In this application, the first model includes a first sub-model and a second sub-model, and the second sub-model includes a third sub-model and a fourth sub-model.

[0126] In some embodiments, the first sub-model may be a backbone network, the second sub-model may be a pose head net, the third sub-model may be a two-dimensional pose head net in the pose head net, and the fourth sub-model may be a three-dimensional pose head net in the pose head net.

[0127] The backbone network can be any existing backbone network. For example, the backbone network can be VGGNet, ResNet, DenseNet, or SENet. Taking ResNet as an example, the network structure of ResNet improves information flow by introducing residual connections, enabling more efficient gradient backpropagation. The pose head network can use any existing feature decoding method (for example, multi-layer deconvolution) to process the features encoded by the backbone network. In addition, the 3D pose head network can fuse and use some features from the 2D pose head network.

[0128] In the above step 1004 , the network parameters of the first model may be randomly set to obtain initial values ​​of the model parameters of the first model.

[0129] Step 1005: Input a batch of training images (ie, an example of the first batch of training images) from the M training images into the first model to obtain a posture sequence corresponding to the batch of training images.

[0130] In step 1005, the pose sequences corresponding to the batch of training images include a first type of two-dimensional pose sequence corresponding to the batch of training images, a first type of three-dimensional pose sequence corresponding to the batch of training images, a common two-dimensional pose sequence corresponding to the batch of training images, and a common three-dimensional pose sequence corresponding to the batch of training images. The first type of two-dimensional pose sequence corresponding to the batch of training images is obtained based on a two-dimensional heat map of the characters included in the batch of training images, the first type of three-dimensional pose sequence corresponding to the batch of training images is obtained based on a three-dimensional heat map of the characters included in the batch of training images, the common two-dimensional pose sequence is determined based on pose key points shared by the two-dimensional pose sequence corresponding to the batch of training images and the three-dimensional pose sequence corresponding to the batch of training images, and the common three-dimensional pose sequence is determined based on pose key points shared by the two-dimensional pose sequence corresponding to the batch of training images and the three-dimensional pose sequence corresponding to the batch of training images.

[0131] As mentioned above, the first type of two-dimensional posture sequence corresponding to a batch of training images is obtained based on the two-dimensional heat map of the characters included in the batch of training images, and specifically may include:

[0132] The center of gravity of the two-dimensional posture key points included in the two-dimensional heat map of the characters included in a batch of training images is calculated to obtain a first type of two-dimensional posture sequence.

[0133] As mentioned above, the first type of 3D posture sequence corresponding to a batch of training images is obtained based on the 3D heat map of the characters included in the batch of training images, and specifically may include:

[0134] The center of gravity of the three-dimensional posture key points included in the three-dimensional heat map of the characters included in a batch of training images is calculated to obtain a first type of three-dimensional posture sequence.

[0135] As an example but not limited to, when the above-mentioned batch of training images is T (T is an integer less than or equal to M and greater than 0) training images among the above-mentioned M training images, specifically, the posture sequence corresponding to the above-mentioned batch of training images may include: a two-dimensional posture sequence (i.e., an example of the first type of two-dimensional pose sequence corresponding to the above batch of training images), three-dimensional pose sequence (i.e., an example of the first type of three-dimensional posture sequence corresponding to the above batch of training images), common two-dimensional posture sequence and common 3D pose sequences Among them, P i 2d is the first type of two-dimensional posture of the person included in the i-th training image in the above batch of training images; P i 3d is the first type of 3D posture of the person included in the i-th training image in the above batch of training images; is the common two-dimensional posture of the person included in the i-th training image in the above batch of training images; is the common 3D posture of the person included in the i-th training image in the above batch of training images, and i=1,2,…,T.

[0136] It should be noted that the common two-dimensional posture is based on the two-dimensional posture P i 2d and 3D pose P i 3d Obtained. Common 3D pose According to the two-dimensional posture P i 2d and 3D pose P i 3d Obtained.

[0137] As an example and not limitation, the following steps can be performed based on the two-dimensional posture P i 2d and 3D pose P i 3d Get the common two-dimensional pose and common 3D posture

[0138] The key points of the 2D pose are 2d_joints_name = ['Nose','L_Eye','R_Eye','L_Ear','R_Ear','L_Shoulder','R_Shoulder','L_Elbow','R_Elbow','L_Wrist','R_Wrist','L_Hip','R_Hip','L_Knee','R_Knee','L_Ankle','R_Ankle','Thorax','Pelvis'] ]

[0140] The key points of the three-dimensional posture are 3d_joints_name = ['Pelvis','R_Hip','R_Knee','R_Ankle','L_Hip','L_Knee','L_Ankle','Torso','Neck','Nose','Head','L_Shoulder','L_Elbow','L_Wrist','R_Shoulder','R_Elbow','R_Wrist','Thorax' ]

[0142] The common key points are common_joints_name = ['R_Hip','R_Knee','R_Ankle','L_Hip','L_Knee','L_Ankle','Nose','L_Shoulder','L_Elbow','L_Wrist','R_Shoulder','R_Elbow','R_Wrist','Thorax' ]

[0144] The key point conversion index of the common two-dimensional posture can be obtained as 2d_joint_index = [12, 14, 16, 11, 13, 15, 0, 5, 7, 9, 6, 8, 10, 17].

[0145] The index value at the i-th position in 2d_joint_index represents the index of the posture key point at the i-th position in the common key point among the key points of the two-dimensional posture, and i is an integer greater than or equal to 0 and less than or equal to 13.

[0146] For example, "12" can be understood as the index of 'R_Hip' in the common keypoints among the keypoints of the 2D pose. Similarly, "14" can be understood as the index of 'R_Knee' in the common keypoints among the keypoints of the 2D pose.

[0147] The key point conversion index of the common three-dimensional posture can be obtained as 3d_joint_index = [1, 2, 3, 4, 5, 6, 9, 11, 12, 13, 14, 15, 16, 17].

[0148] The index value at the i-th position in 3d_joint_index represents the index of the posture key point at the i-th position in the common key point among the key points of the three-dimensional posture, and i is an integer greater than or equal to 0 and less than or equal to 13.

[0149] For example, “1” can be understood as the index of 'R_Hip' in the common key point in the key points of the two-dimensional pose.

[0150] Therefore, the corresponding common two-dimensional posture can be obtained as

[0151] Therefore, the corresponding common three-dimensional posture can be obtained as

[0152] The posture heatmaps of the above-mentioned batch of training images include a two-dimensional posture heatmap, a three-dimensional posture heatmap, and a posture key point confidence heatmap of the person included in each training image.

[0153] In this application, there is no specific limit on the number of posture key points included in each heat map.

[0154] In some embodiments, each heatmap may include only one pose keypoint. For example, image A includes a target person with N pose keypoints. In this case, the heatmap corresponding to image A may be N heatmaps, each of which corresponds one-to-one to each of the N pose keypoints.

[0155] In some embodiments, each heatmap may include multiple posture keypoints. For example, the person included in image A includes six posture keypoints. In this case, the heatmap corresponding to image A may be one heatmap. In other words, this one heatmap includes six posture keypoints.

[0156] In this application, the two-dimensional posture and / or three-dimensional posture of the character can be obtained based on the character's posture heat map.

[0157] In some embodiments, the center of gravity of key points included in the character's two-dimensional posture heat map can be calculated using the first sub-model to obtain the character's first type of two-dimensional posture and the character's first type of three-dimensional posture.

[0158] In this application, there is no specific limitation on the key points of character posture.

[0159] like Figure 4As shown in the figure, the head, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle are all key points of the character's posture. In other words, Figure 4 Thirteen character pose key points are shown, and these character pose key points are numbered from (1) to (13).

[0160] It should be understood that the key points of character posture in this application are not limited to the above-mentioned key points of posture, and the above-mentioned key points of character posture can also be called human joints.

[0161] Step 1006: Determine the loss value of the first model based on the posture sequence corresponding to the above batch of training images.

[0162] The loss value of the first model in the above step 1006 is obtained by weighting the first loss value, the second loss value, the third loss value and the fourth loss value.

[0163] Among them, the first loss value is determined based on the first type of two-dimensional posture sequence corresponding to a batch of training images and the two-dimensional posture labels corresponding to a batch of training images, the second loss value is determined based on the first type of three-dimensional posture sequence corresponding to a batch of training images and the three-dimensional posture labels corresponding to a batch of training images, the third loss value is determined based on the common two-dimensional posture sequence corresponding to a batch of training images and the common three-dimensional posture sequence corresponding to a batch of training images, and the fourth loss value is determined based on the posture key point confidence heat map labels corresponding to a batch of training images and the posture key point confidence heat map corresponding to a batch of training images.

[0164] For example, the first loss value, the second loss value, the third loss value, or the fourth loss value may be calculated according to a distance formula.

[0165] Step 1007: Update the model parameters of the first model according to the loss value of the first model.

[0166] Step 1008: Determine whether the first model meets the preset requirements.

[0167] Optionally, the first model meets the preset requirements, including: the first model meets at least one of the following conditions:

[0168] (1) The number of training times of the first model is greater than or equal to the preset number;

[0169] (2) The loss value of the first model is less than or equal to the preset threshold.

[0170] Optionally, in some implementations, the loss value of the first model is less than or equal to a preset threshold value, which may also be replaced by a change in the loss value of the first model is less than or equal to a preset threshold value.

[0171] In step 1008, when the two-dimensional and three-dimensional pose joint estimation network satisfies at least one of the above conditions (1) or (2), it can be determined that the first model meets the preset requirements, and step 1009 is executed to end the training process of the first model; and when the first model does not meet any of the above conditions (1) or (2), it means that the first model has not yet met the preset requirements and needs to continue to train the first model, that is, re-execute steps 1005 to 1008 until a first model that meets the preset requirements is obtained. The above preset threshold can be flexibly set based on experience. When the preset threshold is set too large, the output effect of the first model obtained by training may not be good enough, and when the preset threshold is set too small, the loss value of the first model may be difficult to converge during training.

[0172] Optionally, the preset threshold value has a value range of [0, 0.1].

[0173] Specifically, the value of the preset threshold may be 0.01.

[0174] Step 1009, end.

[0175] The above step 1009 indicates the end of the training process of the first model.

[0176] It should be understood that in the embodiment of the present application, after the image sequence of the person to be identified is processed by the third sub-model in the trained first model, a first-category two-dimensional pose sequence corresponding to the image sequence of the person to be identified can be obtained. After the image sequence of the person to be identified is processed by the fourth sub-model in the trained first model, a first-category three-dimensional pose sequence corresponding to the image sequence of the person to be identified can be obtained. After this, the first model can also perform the following steps:

[0177] The first model rotates the first type of three-dimensional posture sequence corresponding to the image sequence of the person to be identified at any or specific angle to obtain a second type of three-dimensional posture sequence corresponding to the image sequence of the person to be identified;

[0178] The first model performs a two-dimensional plane projection on the second type of three-dimensional posture sequence corresponding to the image sequence of the person to be identified, to obtain the second type of two-dimensional posture sequence corresponding to the image sequence of the person to be identified.

[0179] The second type of 2D pose is different from the first type of 2D pose. This can be understood as the two types of 2D poses having different perspectives. In other words, the two types of 2D poses are obtained by shooting from different perspectives.

[0180] As an example and not limitation, when the first type of 3D pose sequence can be expressed as T represents the total number of images of people to be identified, P1 3d Represents the first type of 3D posture sequence corresponding to the first image of the person to be identified, and so on. Rotate any (or specific) angle around the x-axis to convert the first type of three-dimensional posture sequence The second type of two-dimensional posture sequence can be obtained by rotating an arbitrary (or specific) angle around the y-axis and performing a two-dimensional projection on the xy plane.

[0181] In some implementations, the t-th first-category 3D pose P among the T first-category 3D poses is t 3d (Corresponding to the tth person image to be identified) After rotating around the x-axis, y-axis, and z-axis by angles α, β, and γ respectively, the second type of three-dimensional posture corresponding to the tth person image to be identified is obtained It can be expressed as:

[0182]

[0183] Where R is P t 3d The perspective transformation matrix after rotating α around the x-axis, β around the y-axis, and γ around the z-axis, R x P t 3d The perspective transformation matrix after rotating α around the x-axis, R y P t 3d The perspective transformation matrix after rotating β around the y-axis, R z P t 3d The viewing angle transformation matrix after rotating γ around the z-axis.

[0184] The above R can be expressed as:

[0185]

[0186] Then, the second type of 3D posture corresponding to the t-th person image to be identified is By performing two-dimensional plane projection, we can obtain the second type of two-dimensional posture corresponding to the t-th person image to be identified

[0187] In this application, the angles α, β, and γ are not specifically limited. They can be selected based on specific application scenarios or user needs. For example, α can be 60°, β can be 30°, and γ can be 20°. For example, α can be 20°, β can be 15°, and γ can be 120°.

[0188] An embodiment of the present application provides a method for training a first model, which can predict the posture of a person included in an image to be identified. The first model is trained by using a two-dimensional posture sequence of the person and a three-dimensional posture sequence of the person, and the two-dimensional posture sequence of the person and the three-dimensional posture sequence of the person have a consistent connection. Since the third sub-model in the first model takes into account the three-dimensional posture information of the person's posture when predicting the two-dimensional posture of the person, the prediction result of the third sub-model for predicting the two-dimensional posture of the person has higher accuracy and lower prediction delay. In addition, the first model can also obtain an extended two-dimensional posture sequence (i.e., a second type of two-dimensional posture sequence) based on the three-dimensional posture sequence output by it (i.e., a first type of three-dimensional posture sequence), thereby effectively expanding the sample capacity of the two-dimensional posture sequence.

[0189] Figure 5 It is a schematic flowchart of the training method 200 of the second model in the action recognition model provided in this application.

[0190] like Figure 5 As shown, the method 200 includes steps 2001 to 2008, and steps 2001 to 2008 are described in detail below.

[0191] Step 2001, start.

[0192] The above step 2001 indicates the start of the training process of the second model.

[0193] Step 2002: Obtain training data.

[0194] The training data in step 2002 includes M (M is an integer greater than 1) training images and annotation data for the M training images. Each of the M training images includes a person, and the annotation data for each training image includes a bounding box containing the person in each training image and annotation information about the person's posture in each training image.

[0195] In this application, the character posture annotation information is not specifically limited. For example, the character posture annotation information can be standing, falling, or climbing.

[0196] Step 2003: Input each training image into the first model to obtain the posture of the person included in each training image.

[0197] The first model in step 2003 is a model obtained after training according to method 100. The first model used in step 2003 can be trained according to the training method of the first model of the embodiment of the present application. Specifically, the first model can be obtained by Figure 3 The method shown is trained.

[0198] The pose of the person included in each training image in step 2003 includes: the first-category 2D pose of the person, the second-category 2D pose of the person, and the confidence of the first-category 2D pose. The confidence of the first-category 2D pose of the person is obtained by the first model based on the first-category 2D pose of the person, the first-category 3D pose of the person, and the confidence heat map of the key points of the first-category 2D pose of the person.

[0199] Specifically, when there are T training images, the pose sequences of the characters included in the T training images include: the first type of two-dimensional pose sequence The second type of two-dimensional pose sequence And the first type of two-dimensional posture confidence sequence C={C1,C2,......,C T}. Among them, P i 2d is the first type of two-dimensional posture of the person included in the i-th action image to be recognized among the T action images to be recognized; is the second type of two-dimensional posture of the person included in the i-th action image to be identified among the T action images to be identified; C i is the first type of two-dimensional posture confidence of the person included in the i-th action image to be identified among the T action images to be identified; and i=1,2,…,T. Among them, the first type of two-dimensional posture confidence sequence C={C1,C2,......,C T} is the first model based on the first type of two-dimensional pose sequence The first type of 3D pose sequence And the first type of 2D pose sequence key point confidence heat map Got it.

[0200] Step 2004: Initialize the model parameters of the second model to obtain initial values ​​of the model parameters of the second model.

[0201] In this application, the type of the second model is not specifically limited. For example, the second model can be a convolutional neural network model. For example, the second model can be a recurrent neural network (RNN) model.

[0202] In the above step 2004 , the network parameters of the action recognition network may be randomly set to obtain initial values ​​of the network parameters of the action recognition network.

[0203] Step 2005: construct the posture sequence data of the training image to obtain a feature map.

[0204] In the embodiment of the present application, the posture sequence data of the training image is constructed to obtain a feature map, which may specifically include:

[0205] The two-dimensional pose sequences corresponding to a batch of training images (i.e., an example of the second batch of training images) in the M training images and the key point confidences of the first type of two-dimensional pose sequences corresponding to the batch of training images are spliced ​​to obtain a feature map.

[0206] For example, when the batch of training images is P (P is an integer less than or equal to M and greater than or equal to 1), and the first-category two-dimensional pose sequence corresponding to the P training images includes P first-category two-dimensional poses, the two-dimensional coordinates of each key point in each of the P first-category two-dimensional poses and the corresponding first-category two-dimensional pose key point confidence score can be concatenated as the feature of the pose key point, thereby obtaining a feature map of dimension D×T×N. Here, D is the feature dimension of each first-category two-dimensional pose key point, T is the number of sequence frames, and N is the number of pose key points of the person included in the T-th training image among the P training images.

[0207] Optionally, in some embodiments, constructing posture sequence data of training images to obtain feature maps may further include the following steps:

[0208] A batch of first-category two-dimensional poses corresponding to M training images, a batch of second-category two-dimensional poses corresponding to M training images, and a batch of two-dimensional pose key point confidences corresponding to M training images are spliced ​​to obtain a feature map.

[0209] For example, in the first type of two-dimensional pose sequence corresponding to M training images The second type of 2D pose sequence corresponding to M training images T two-dimensional pose sequences are randomly selected from the dataset, where the T two-dimensional pose sequences include T1 first-category two-dimensional pose sequences and T2 second-category two-dimensional pose sequences, where T1 and T2 are integers greater than or equal to zero, and 0≤T1+T2≤T; and the T1 first-category two-dimensional pose sequences, T2 second-category two-dimensional pose sequences, and T1 first-category two-dimensional pose key point confidences are concatenated to obtain a feature map.

[0210] Step 2006: Determine the loss value of the second model based on the above-mentioned characteristic graph.

[0211] Step 2007: Determine whether the second model meets the preset requirements.

[0212] Optionally, the second model network meets preset requirements, including: the second model meets at least one of the following conditions:

[0213] (1) The number of training times of the second model is greater than or equal to the preset number;

[0214] (2) The loss value of the second model is less than or equal to the preset threshold.

[0215] Optionally, in some implementations, the loss value of the second model is less than or equal to a preset threshold value, and alternatively, the change in the loss value of the second model is less than or equal to a preset threshold value.

[0216] In step 2007, when the second model satisfies at least one of the above conditions (1) or (2), it can be determined that the second model meets the preset requirements, step 2007 is executed, and the training process of the second model ends; when the second model does not meet any one of the above conditions (1) or (2), it means that the second model has not yet met the preset requirements, and it is necessary to continue training the second model, that is, re-execute steps 2005 to 2007 until the second model that meets the preset requirements is obtained.

[0217] The above-mentioned preset threshold can be flexibly set based on experience. When the preset threshold is set too large, the posture recognition effect of the trained action recognition network may not be good enough, and when the preset threshold is set too small, the function value of the loss function may be difficult to converge during training.

[0218] Optionally, the preset threshold value has a value range of [0, 0.01].

[0219] Specifically, the value of the preset threshold may be 0.001.

[0220] Step 2008, end.

[0221] The above step 2008 indicates the end of the training process of the second model.

[0222] In an embodiment of the present application, a method for training a second model is provided. Specifically, the second model uses the two-dimensional posture sequence output by the first model and the two-dimensional posture sequence expanded from the three-dimensional posture output by the first model to train the second model. Using the second model provided in the embodiment of the present application, accurate and rapid recognition of small sample human posture actions can be achieved. Specifically, using the two-dimensional posture sequence expanded from the three-dimensional posture output by the first model and the two-dimensional posture sequence output by the first model as training data for the second model can effectively improve the diversity of data and the generalization of the model, thereby improving the accuracy and recognition efficiency of action recognition in small sample action recognition scenarios.

[0223] Figure 6 It is a schematic flow chart of the action recognition method 300 according to an embodiment of the present application.

[0224] like Figure 6 As shown, the method 300 includes steps 3001 to 3004, and steps 3001 to 3004 are described in detail below.

[0225] Step 3001: Obtain a sequence of images of people to be identified.

[0226] Step 3002: Use the first model to process the image sequence of the person to be identified to obtain a two-dimensional posture sequence corresponding to the image sequence of the person to be identified and the key point confidence of the two-dimensional posture sequence. The two-dimensional posture sequence is obtained based on the two-dimensional heat map of the person included in the image sequence of the person to be identified.

[0227] The first model includes a third sub-model, and the first model can be Figure 3 The method 100 shown is trained.

[0228] In the embodiment of the present application, the first model is used to process the image sequence of the person to be identified to obtain the two-dimensional posture sequence and the key point confidence of the two-dimensional posture sequence corresponding to the image sequence of the person to be identified, which may specifically include:

[0229] The image sequence of the person to be identified is processed by the third sub-model to obtain a two-dimensional posture sequence corresponding to the image sequence of the person to be identified and the confidence of the key points of the two-dimensional posture sequence.

[0230] Step 3003: Concatenate the two-dimensional posture sequence and the key point confidence of the two-dimensional posture sequence to obtain a feature map. The key point confidence of the two-dimensional posture sequence is determined based on the two-dimensional posture sequence and the posture key point heat map corresponding to the two-dimensional posture sequence.

[0231] Specifically, the method for constructing the feature map corresponding to the image to be identified is the same as the method in step 2006 and will not be described in detail here.

[0232] Step 3004: Input the feature map into the second model to obtain the action recognition result of the image sequence of the person to be recognized.

[0233] The second model in step 3004 can be Figure 5 The method 200 shown is trained.

[0234] The recognition result in the above step 3004 includes the action category of the target person included in the image to be recognized, the accuracy of action recognition, the warning flag, and the visualization result.

[0235] The following describes the effect of the action recognition network of the embodiment of the present application in combination with specific experiments and experimental results.

[0236] The experiment sets up four action categories, for example, these four action categories are falling, climbing, standing and squatting.

[0237] For each action category, 40 action sequences extracted from online videos are used as training samples, and 50-300 action sequences extracted from surveillance videos are used as testing samples.

[0238] The accuracy of action recognition is quantitatively measured by using the average accuracy (Average_Acc) indicator. Specifically, the average accuracy (Average_Acc) can be calculated using the following formula:

[0239]

[0240] Where C is the number of action categories, here C = 4; N i is the number of sequence samples of the i-th action category; is the true action category label of the jth sequence of the i-th action category; is the predicted action category of the jth sequence of the i-th action category; Indicates when and When equal, Equal to 1, otherwise Equal to zero.

[0241] It should be understood that the higher the accuracy of action recognition, the larger the corresponding average accuracy (Average_Acc) value will be.

[0242] Table 1

[0243] Training data Test data Average accuracy Existing plan 1 2D Pose 2D Pose 65.47％ Existing Plan 2 3D Pose (w / o consist) 2D Pose 59.57％ Existing plan 3 3D Pose (w / consist) 2D Pose 66.50％ This application plan 3D Pose (w / consist, w / aug) 2D Pose 72.59％

[0244] Table 1 shows the results of different solutions tested using the same test data, where the test results include the average accuracy (Average_Acc). 3D Pose indicates that the training data is two-dimensional projection data of three-dimensional poses. 2D Pose indicates that the test data uses two-dimensional pose data. w / consist indicates that the three-dimensional pose data used in training is consistent with the two-dimensional pose data used in testing. w / o consist indicates that the three-dimensional pose data used in training is inconsistent with the two-dimensional pose data used in testing. w / aug indicates that three-dimensional pose viewpoint augmentation data was used during training.

[0245] As can be seen from Table 1, the average accuracy (Average_Acc) of the proposed solution is higher than that of the existing solution, and has a better recognition effect.

[0246] Combined with the above Figures 1 to 6 The training method of the action recognition model and the action recognition method of the embodiment of the present application are introduced in detail. Figures 7 to 10The training device for the action recognition model and the action recognition device provided in the embodiments of the present application are introduced.

[0247] Figure 7 It is a schematic block diagram of a training device 7000 for a motion recognition model according to an embodiment of the present application. Figure 7 The training device 7000 of the action recognition model shown includes an acquisition unit 7001, a processing unit 7002 and a training unit 7003.

[0248] The acquisition unit 7001 is configured to acquire M training images and annotation data for the M training images, wherein each of the M training images includes a person, and the annotation data for each training image includes a bounding box containing the person in each training image and annotation information about the person's posture in each training image, where M is an integer greater than 1;

[0249] The processing unit 7002 processes the M training images through the first model to obtain posture sequences corresponding to the M training images, where the posture sequences corresponding to the M training images include a first type of two-dimensional posture sequence corresponding to the M training images and a second type of two-dimensional posture sequence corresponding to the M training images, the first type of two-dimensional posture sequence corresponding to the M training images is obtained based on a two-dimensional heat map of the person included in the M training images, the second type of two-dimensional posture sequence corresponding to the M training images is obtained by performing a two-dimensional projection on the second type of three-dimensional posture sequence corresponding to the M training images, the second type of three-dimensional posture sequence corresponding to the M training images is obtained by rotating the first type of three-dimensional posture sequence corresponding to the M training images by an arbitrary or specific angle, and the first type of three-dimensional posture sequence corresponding to the M training images is obtained based on the three-dimensional heat map of the person included in the M training images;

[0250] The training unit 7003 trains the second model by using the posture sequences corresponding to the M training images until the second model meets the second preset requirement;

[0251] The training unit 7003 is further configured to train the first model through the following steps:

[0252] Step 1: Process the person pose annotation information in each training image to obtain the person pose label in each training image, where the pose label includes a two-dimensional pose label, a three-dimensional pose label, and a two-dimensional pose key point confidence heat map label;

[0253] Step 2: Initializing the first model to obtain initial values ​​of model parameters of the first model;

[0254] Step 3: inputting the first batch of training images among the M training images into the first model to obtain a posture sequence corresponding to the first batch of training images, wherein the posture sequence corresponding to the first batch of training images includes a first type of two-dimensional posture sequence corresponding to the first batch of training images, a first type of three-dimensional posture sequence corresponding to the first batch of training images, a common two-dimensional posture sequence corresponding to the first batch of training images, and a common three-dimensional posture sequence corresponding to the first batch of training images, wherein the first type of two-dimensional posture sequence corresponding to the first batch of training images is obtained based on the two-dimensional heat map of the characters included in the first batch of training images, the first type of three-dimensional posture sequence corresponding to the first batch of training images is obtained based on the three-dimensional heat map of the characters included in the first batch of training images, the common two-dimensional posture sequence is a two-dimensional posture sequence determined based on the common posture key points in the two-dimensional posture sequence corresponding to the first batch of training images and the three-dimensional posture sequence corresponding to the first batch of training images, and the common three-dimensional posture sequence is a three-dimensional posture sequence determined based on the common posture key points in the two-dimensional posture sequence corresponding to the first batch of training images and the three-dimensional posture sequence corresponding to the first batch of training images;

[0255] Step 4: Determine the loss value of the first model based on the posture sequence corresponding to the first batch of training images. The loss value of the first model is obtained by weighted processing of the first loss value, the second loss value, the third loss value, and the fourth loss value. The first loss value is determined based on the first type of two-dimensional posture sequence corresponding to the first batch of training images and the two-dimensional posture labels corresponding to the first batch of training images. The second loss value is determined based on the first type of three-dimensional posture sequence corresponding to the first batch of training images and the three-dimensional posture labels corresponding to the first batch of training images. The third loss value is determined based on the common two-dimensional posture sequence corresponding to the first batch of training images and the common three-dimensional posture sequence corresponding to the first batch of training images. The fourth loss value is determined based on the two-dimensional posture key point confidence heat map labels corresponding to the first batch of training images and the two-dimensional posture key point confidence heat map corresponding to the first batch of training images.

[0256] Step 6: Update the model parameters of the first model according to the loss value of the first model;

[0257] Repeat steps 3 to 5 above until the first model meets the first preset requirement.

[0258] Optionally, in some embodiments, the first model includes a first sub-model and a second sub-model, the second sub-model includes a third sub-model and a fourth sub-model, and the processing unit 7002 is further configured to:

[0259] Processing the M training images by the third sub-model to obtain a first type of two-dimensional posture sequence corresponding to the M training images;

[0260] Processing the M training images by the fourth sub-model to obtain a first type of three-dimensional pose sequence corresponding to the M training images;

[0261] The first model is used to rotate the first type of three-dimensional posture sequence corresponding to the M training images at any or specific angle to obtain the second type of two-dimensional posture sequence corresponding to the M training images.

[0262] Optionally, in some embodiments, the training unit 7003 is further configured to train the second model through the following steps:

[0263] Step 7: Initializing the model parameters of the second model to obtain initial values ​​of the model parameters of the second model;

[0264] Step 8: splicing the key point confidences of the two-dimensional pose sequences corresponding to the second batch of training images in the M training images and the first type of two-dimensional pose sequences corresponding to the second batch of training images to obtain a feature map, wherein the key point confidences of the first type of two-dimensional pose sequences corresponding to the second batch of training images are determined by the first model based on the first type of two-dimensional pose sequences corresponding to the second batch of training images, the first type of three-dimensional pose sequences corresponding to the second batch of training images, and the pose key point heat map corresponding to the first type of two-dimensional pose sequences corresponding to the second batch of training images;

[0265] Step 9: Input the feature map into the second model to obtain the loss value of the second model;

[0266] Step 10: updating the model parameters of the second model according to the loss value of the second model;

[0267] Repeat steps 8 to 10 until the second model meets the second preset requirement.

[0268] Optionally, in some embodiments, the processing unit 7002 is further configured to:

[0269] splicing the key point confidences of the second type of two-dimensional pose sequence corresponding to the second batch of training images and the first type of two-dimensional pose sequence corresponding to the second batch of training images to obtain the feature map; or

[0270] The first type of two-dimensional posture sequence corresponding to the second batch of training images, the second type of two-dimensional posture sequence corresponding to the second batch of training images, and the key point confidence of the first type of two-dimensional posture sequence corresponding to the second batch of training images are spliced ​​to obtain the feature map.

[0271] Optionally, in some embodiments, the second model satisfies a second preset requirement, including:

[0272] The second model meets the second preset requirement when at least one of the following conditions is met:

[0273] The number of training times of the second model is greater than or equal to a second preset number;

[0274] The loss value of the second model is less than or equal to a second preset threshold.

[0275] Optionally, in some embodiments, the first model satisfies the first preset requirement, including:

[0276] The first model meets the first preset requirement when at least one of the following conditions is met:

[0277] The number of training times of the first model is greater than or equal to a first preset number;

[0278] The loss value of the first model is less than or equal to a first preset threshold.

[0279] Figure 8 8 is a schematic block diagram of a training device 8000 for an action recognition model according to an embodiment of the present application. Figure 8 As shown, the training device 8000 for the action recognition model includes: a transceiver 8010, a processor 8020, and a memory 8030. The transceiver 8010, the processor 8020, and the memory 8030 communicate with each other through internal connection paths to transmit control and / or data signals. The memory 8030 is used to store computer programs, and the processor 8010 is used to call and run the computer programs from the memory 8030 to control the transceiver 8020 to send and receive signals.

[0280] Specifically, the function of the transceiver 8010 is similar to Figure 7 The specific functions of the acquisition unit 7001 shown correspond to those of the embodiment and will not be repeated here.

[0281] Specifically, the functions of processor 8020 are similar to Figure 7 The specific functions of the processing unit 7002 and the training unit 7003 shown correspond to each other and will not be repeated here.

[0282] Figure 9 is a schematic block diagram of the motion recognition device 9000 according to an embodiment of the present application. Figure 9 The motion recognition device 9000 shown includes an acquisition unit 9001 and a processing unit 9002 .

[0283] The acquisition unit 9001 is used to acquire a sequence of images of a person to be identified.

[0284] The processing unit 9002 is configured to process the image sequence of the person to be identified to obtain a two-dimensional posture sequence corresponding to the image sequence of the person to be identified and confidence scores of key points of the two-dimensional posture sequence, wherein the two-dimensional posture sequence is obtained based on a two-dimensional heat map of the person included in the image sequence of the person to be identified;

[0285] The processing unit 9002 is further configured to:

[0286] The two-dimensional posture sequence and the confidence of the key points of the two-dimensional posture sequence are spliced ​​to obtain a feature map, where the confidence of the key points of the two-dimensional posture sequence is determined based on the two-dimensional posture sequence and the posture key point heat map corresponding to the two-dimensional posture sequence;

[0287] The feature map is input into the second model to obtain the action recognition result of the image sequence of the person to be recognized.

[0288] Optionally, in some embodiments, the first model includes a third sub-model, and the processing unit 9002 is further configured to:

[0289] The image sequence of the person to be identified is processed by the third sub-model to obtain a two-dimensional posture sequence corresponding to the image sequence of the person to be identified and confidence scores of key points of the two-dimensional posture sequence.

[0290] Figure 10 1 is a schematic block diagram of the motion recognition device 1000 according to an embodiment of the present application. Figure 10 As shown, the motion recognition device 1000 includes: a transceiver 1010, a processor 1020, and a memory 1030. The transceiver 1010, the processor 1020, and the memory 1030 communicate with each other through internal connection paths to transmit control and / or data signals. The memory 1030 is used to store computer programs, and the processor 1010 is used to call and run the computer programs from the memory 1030 to control the transceiver 1020 to send and receive signals.

[0291] Specifically, the function of the transceiver 1010 is similar to Figure 9 The specific functions of the acquisition unit 910 shown correspond to those of the embodiment and are not described again here.

[0292] Specifically, the function of the processor 1020 is similar to Figure 9 The specific functions of the training unit 920 shown correspond to those shown in the figure and will not be repeated here.

[0293] The chip in the embodiments of the present application can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or other integrated chips.

[0294] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it will not be described in detail here.

[0295] It should be noted that the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiment can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0296] It is understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0297] According to the method provided in the embodiment of the present application, the present application also provides a computer program product, which includes: a computer program code, which, when executed on a computer, causes the computer to execute Figure 3 、 Figure 5 and Figure 6 A method according to any one of the embodiments shown.

[0298] According to the method provided in the embodiment of the present application, the present application also provides a computer-readable medium, which stores a program code, and when the program code is run on a computer, the computer executes Figure 3 、 Figure 5 and Figure 6 A method according to any one of the embodiments shown.

[0299] According to the method provided in the embodiment of the present application, the present application also provides a system, which includes the aforementioned one or more first models and one or more second models.

[0300] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0301] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0302] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0303] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0304] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0305] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0306] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for training an action recognition model, characterized in that: The action recognition model includes a first model and a second model, and the method includes: Obtaining M training images and annotation data for the M training images, wherein each of the M training images includes a person, and the annotation data for each training image includes a bounding box containing the person in each training image and annotation information of the person's posture in each training image, where M is an integer greater than 1; The M training images are processed by the first model to obtain posture sequences corresponding to the M training images, where the posture sequences corresponding to the M training images include a first type of two-dimensional posture sequence corresponding to the M training images and a second type of two-dimensional posture sequence corresponding to the M training images, the first type of two-dimensional posture sequence corresponding to the M training images is obtained based on a two-dimensional heat map of the characters included in the M training images, the second type of two-dimensional posture sequence corresponding to the M training images is obtained by performing two-dimensional projection on the second type of three-dimensional posture sequence corresponding to the M training images, the second type of three-dimensional posture sequence corresponding to the M training images is obtained by rotating the first type of three-dimensional posture sequence corresponding to the M training images by an arbitrary or specific angle, and the first type of three-dimensional posture sequence corresponding to the M training images is obtained based on the three-dimensional heat map of the characters included in the M training images; Training the second model by using the posture sequences corresponding to the M training images until the second model meets a second preset requirement; The first model is trained by the following steps: Step 1: Processing the person pose annotation information in each training image to obtain a person pose label in each training image, wherein the pose label includes a two-dimensional pose label, a three-dimensional pose label, and a two-dimensional pose key point confidence heat map label; Step 2: Initializing the first model to obtain initial values ​​of model parameters of the first model; Step 3: Inputting the first batch of training images among the M training images into the first model to obtain a posture sequence corresponding to the first batch of training images, wherein the posture sequence corresponding to the first batch of training images includes a first type of two-dimensional posture sequence corresponding to the first batch of training images, a first type of three-dimensional posture sequence corresponding to the first batch of training images, a common two-dimensional posture sequence corresponding to the first batch of training images, and a common three-dimensional posture sequence corresponding to the first batch of training images. The first type of two-dimensional posture sequence corresponding to the first batch of training images is obtained based on the two-dimensional heat map of the characters included in the first batch of training images. The first type of three-dimensional posture sequence corresponding to the first batch of training images is obtained based on the three-dimensional heat map of the characters included in the first batch of training images. The common two-dimensional posture sequence is a two-dimensional posture sequence determined based on the common posture key points in the two-dimensional posture sequence corresponding to the first batch of training images and the three-dimensional posture sequence corresponding to the first batch of training images. The common three-dimensional posture sequence is a three-dimensional posture sequence determined based on the common posture key points in the two-dimensional posture sequence corresponding to the first batch of training images and the three-dimensional posture sequence corresponding to the first batch of training images. Step 4: Determine the loss value of the first model based on the posture sequence corresponding to the first batch of training images, where the loss value of the first model is obtained by weighted processing of the first loss value, the second loss value, the third loss value, and the fourth loss value. The first loss value is determined based on the first type of two-dimensional posture sequence corresponding to the first batch of training images and the two-dimensional posture labels corresponding to the first batch of training images. The second loss value is determined based on the first type of three-dimensional posture sequence corresponding to the first batch of training images and the three-dimensional posture labels corresponding to the first batch of training images. The third loss value is determined based on the common two-dimensional posture sequence corresponding to the first batch of training images and the common three-dimensional posture sequence corresponding to the first batch of training images. The fourth loss value is determined based on the two-dimensional posture key point confidence heat map labels corresponding to the first batch of training images and the two-dimensional posture key point confidence heat map corresponding to the first batch of training images. Step 6: Update the model parameters of the first model according to the loss value of the first model; Repeat steps 3 to 5 above until the first model meets the first preset requirement.

2. The training method according to claim 1, wherein: The first model includes a first sub-model and a second sub-model, the second sub-model includes a third sub-model and a fourth sub-model, and the processing of the M training images by the first model to obtain posture sequences corresponding to the M training images includes: Processing the M training images by using the third sub-model to obtain a first type of two-dimensional posture sequence corresponding to the M training images; Processing the M training images by using the fourth sub-model to obtain a first type of three-dimensional posture sequence corresponding to the M training images; The first model is used to rotate the first type of three-dimensional posture sequence corresponding to the M training images at any or specific angle to obtain the second type of two-dimensional posture sequence corresponding to the M training images.

3. The training method according to claim 1 or 2, characterized in that: The second model is trained by the following steps: Step 7: Initializing the model parameters of the second model to obtain initial values ​​of the model parameters of the second model; Step 8: splicing the key point confidences of the two-dimensional pose sequences corresponding to the second batch of training images in the M training images and the first type of two-dimensional pose sequences corresponding to the second batch of training images to obtain a feature map, wherein the key point confidences of the first type of two-dimensional pose sequences corresponding to the second batch of training images are determined by the first model based on the first type of two-dimensional pose sequences corresponding to the second batch of training images, the first type of three-dimensional pose sequences corresponding to the second batch of training images, and the pose key point heat map corresponding to the first type of two-dimensional pose sequences corresponding to the second batch of training images; Step 9: Input the feature map into the second model to obtain the loss value of the second model; Step 10: updating the model parameters of the second model according to the loss value of the second model; Repeat steps 8 to 10 above until the second model meets the second preset requirement.

4. The training method according to claim 3, wherein: The step of splicing the key point confidences of the two-dimensional pose sequences corresponding to the second batch of training images in the M training images and the first type of two-dimensional pose sequences corresponding to the second batch of training images to obtain a feature map includes: splicing the key point confidences of the second type of two-dimensional pose sequence corresponding to the second batch of training images and the first type of two-dimensional pose sequence corresponding to the second batch of training images to obtain the feature map; or The key point confidences of the first type of two-dimensional pose sequence corresponding to the second batch of training images, the second type of two-dimensional pose sequence corresponding to the second batch of training images, and the first type of two-dimensional pose sequence corresponding to the second batch of training images are spliced ​​to obtain the feature map.

5. The training method according to claim 1 or 2, characterized in that: The second model meets the second preset requirement, including: When at least one of the following conditions is met, the second model meets the second preset requirement: The number of training times of the second model is greater than or equal to a second preset number; The loss value of the second model is less than or equal to a second preset threshold.

6. The training method according to claim 1 or 2, characterized in that: The first model meets the first preset requirement, including: When at least one of the following conditions is met, the first model meets the first preset requirement: The number of training times of the first model is greater than or equal to a first preset number; The loss value of the first model is less than or equal to a first preset threshold.

7. A method for motion recognition, characterized in that: The method comprises: Obtain a sequence of images of people to be identified; Processing the image sequence of the person to be identified using the first model to obtain a two-dimensional pose sequence corresponding to the image sequence of the person to be identified and key point confidences of the two-dimensional pose sequence, wherein the two-dimensional pose sequence is obtained based on a two-dimensional heat map of the person included in the image sequence of the person to be identified; splicing the two-dimensional pose sequence and the key point confidence of the two-dimensional pose sequence to obtain a feature map, wherein the key point confidence of the two-dimensional pose sequence is determined based on the two-dimensional pose sequence and the pose key point heat map corresponding to the two-dimensional pose sequence; Inputting the feature map into a second model to obtain an action recognition result of the image sequence of the person to be recognized; The first model and the second model are trained according to the training method according to any one of claims 1 to 6.

8. The identification method according to claim 7, wherein: The first model includes a third sub-model, and the first model is used to process the image sequence of the person to be identified to obtain a two-dimensional posture sequence corresponding to the image sequence of the person to be identified and the confidence of the key points of the two-dimensional posture sequence, including: The image sequence of the person to be identified is processed by the third sub-model to obtain a two-dimensional posture sequence corresponding to the image sequence of the person to be identified and confidences of key points of the two-dimensional posture sequence.

9. A training device for an action recognition model, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions to execute the training method according to any one of claims 1 to 6.

10. A motion recognition device, characterized in that: It comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions to execute the action recognition method according to claim 7 or 8.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program code for execution by a device, wherein the program code includes instructions for executing the training method according to any one of claims 1 to 6.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program code for execution by a device, wherein the program code includes instructions for executing the motion recognition method according to claim 7 or 8.

13. A chip, characterized in that: The chip includes a processor and a data interface, and the processor reads instructions stored in a memory through the data interface to execute the training method according to any one of claims 1 to 6.

14. A chip, characterized in that: The chip includes a processor and a data interface, and the processor reads instructions stored in the memory through the data interface to execute the action recognition method according to claim 7 or 8.

Images