A gesture recognition method and related device

Abstract

This application provides a pose recognition method and related equipment. The pose of the target object obtained after processing an image containing the target object has high accuracy, thereby improving the accuracy of the motion information of the acquired target image. The method includes: acquiring a target image; processing the target region of the target image and the position information of the target region in the target image using a pose recognition model to obtain pose parameters, where the target region is the area where the target object is located, and the pose parameters are used to obtain the pose of the target object.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a posture recognition method and related equipment. Background Technology

[0002] Human motion capture technology is commonly used in the film, television, and gaming industries. This technology processes input video streams to capture the posture of the human body in each frame, thereby obtaining the body's motion information. Based on this technology, posture data of human movement can be extracted from massive amounts of video, offering a wide range of applications.

[0003] Currently, human motion capture can be achieved using AI technology. Specifically, for a given image, a trained neural network can process the region containing the human body in the image to obtain pose parameters, which can then be used to determine the human body's posture.

[0004] If there are multiple images depicting the same environment, and a human body is located in different positions within that environment in each image, and the human body's posture is extremely similar, the neural network will determine that the posture parameters of the human body in these multiple images are consistent. However, in a real environment, once the human body moves, although the posture of the human body in different positions is similar, there will definitely be subtle differences between the postures. This is something that current neural networks cannot recognize, resulting in inaccurate capture of the human body's posture, which in turn affects the accuracy of the human body's motion information. Summary of the Invention

[0005] This application provides a posture recognition method and related device, which obtains the posture of the target object after processing an image containing the target object with high accuracy, thereby improving the accuracy of the motion information of the acquired target image.

[0006] A first aspect of this application provides a pose recognition method, the method comprising:

[0007] When pose recognition is required for a target image, the target image can be obtained first. The target image usually shows the target object and the environment in which the target object is located. It can be understood that the goal of pose recognition is to obtain the pose of the target object in the target image.

[0008] To reduce the computational load of the pose recognition model, the target image can be preprocessed. Specifically, the target image can be detected to determine the region where the target object is located in the target image, which can be called the target region, and the positional information of the target region in the target image can be obtained.

[0009] After obtaining the target region and its location information within the target image, these can be input into a pose recognition model. The model then processes the target region and its location information to obtain pose parameters. Based on these pose parameters, the pose of the target object in the target image can be determined.

[0010] As can be seen from the above method, after acquiring the target image, the target region where the target object is located and its position information within the target image can be input into the pose recognition model. The pose recognition model then processes the target region and its position information to obtain pose parameters. Therefore, the pose of the target object can be obtained based on these pose parameters. In the aforementioned process, the input to the pose recognition model includes not only the cropped target region but also its position information within the target image. Thus, when processing images, the pose recognition model considers not only the influence of the target region's image information on the target object's pose but also the influence of its position information within the target image. This comprehensive approach results in a more accurate target object pose, thereby improving the accuracy of the motion information obtained from the target image.

[0011] In one possible implementation, the location information includes the coordinates of the center point of the target region in the image coordinate system and the size of the target region, which is constructed based on the target image. In the aforementioned implementation, a vertex of the target image can be used as the origin of the image coordinate system. This allows determination of the coordinates of the center point of the target region in that coordinate system, as well as the length and width of the target region (essentially obtaining its size). This information can be used to indicate the position of the target region within the target image. Therefore, when this location information is input into the pose recognition model, the model can effectively consider the influence of the target region's location information on the pose of the target object during image processing. Consequently, the pose of the target object obtained based on this information has high accuracy.

[0012] In one possible implementation, the location information includes the coordinates of the vertices of the target region in an image coordinate system, which is constructed based on the target image. In the aforementioned implementation, a vertex of the target image can be used as the origin of the image coordinate system. This allows the coordinates of all vertices of the target region in that coordinate system to be determined. This information can be used to indicate the position of the target region in the target image. Therefore, when this location information of the target region is input into the pose recognition model, the model can effectively consider the influence of the target region's location information on the pose of the target object during image processing. Consequently, the pose of the target object obtained based on this information has high accuracy.

[0013] In one possible implementation, the pose recognition model is trained based on the predicted projection of the target object's predicted pose onto the image to be processed and the actual projection of the target object's true pose onto the image to be processed. In this process, during the training of the pose recognition model, the model perceives the target object's position in the entire image and learns the relationship between that position and the target object's pose. Therefore, in the application of the pose recognition model, the model can accurately capture the target object's pose from the entire image based on the target object's position information.

[0014] In one possible implementation, the pose of the target object includes its orientation in the camera coordinate system (i.e., the target object's orientation relative to the camera) and its limb behavior in the camera coordinate system (i.e., the target object's actions in three-dimensional space). The origin of the camera coordinate system is the camera that captured the target image. In the aforementioned implementation, even if multiple images of the target object in different positions within the same environment are obtained, and the target object's pose is similar across these multiple images, the pose recognition model can still identify the differences in the target object's pose between these multiple images, namely, the differences in the target object's orientation relative to the camera, and the differences in the target object's actions in three-dimensional space.

[0015] In one possible implementation, the method further includes: processing the target region of the target image and the position information of the target region in the target image using a pose recognition model to obtain shape parameters and displacement parameters. The pose parameters, shape parameters, and displacement parameters are used together to obtain the pose of the target object. In the aforementioned implementation, the output of the pose recognition model may include pose parameters, shape parameters, and displacement parameters. The pose parameters indicate the rotation angle of the target object relative to the camera and the angles between the joints of the target object itself. The shape parameters indicate the three-dimensional shape of the target object. The displacement parameters indicate the extent to which the target object occupies the target region and the offset of the target object within the target region (e.g., the degree to which the target object shifts to the left or right with the center point of the target region as a reference point). Based on the pose parameters, shape parameters, and displacement parameters, the pose of the target object can be accurately obtained.

[0016] In one possible implementation, the method further includes: normalizing the positional information of the target region in the target image to obtain normalized positional information; and processing the target region of the target image and its positional information in the target image using a pose recognition model to obtain pose parameters, including: processing the target region of the target image and the normalized positional information using a pose recognition model to obtain pose parameters. In the aforementioned implementation, the pose recognition model can perform pose recognition operations based on the normalized positional information. Since the processing of the normalized positional information is less difficult, it helps to reduce the computational load of the pose recognition model and lower the design cost of the model.

[0017] In one possible implementation, the target object is the human body.

[0018] A second aspect of this application provides a pose recognition method, the method comprising:

[0019] When pose recognition is required for a target image, the target image can be obtained first. The target image usually shows the target object and the environment in which the target object is located. It can be understood that the goal of pose recognition is to obtain the pose of the target object in the target image.

[0020] After obtaining the target image, the position information of the pixels in the target image can be obtained. The target image and the position information of the pixels in the target image are then input into the pose recognition model. The pose recognition model processes the target image and the position information of the pixels in the target image to obtain pose parameters. The pose parameters are used to obtain the pose of the target object contained in the target image.

[0021] As can be seen from the above method, after acquiring the target image, the target image and the position information of its pixels can be input into the pose recognition model. The pose recognition model processes the target image and the position information of its pixels to obtain pose parameters. Therefore, the pose of the target object presented in the target image can be obtained based on the pose parameters. In the aforementioned process, the input to the pose recognition model includes not only the target image but also the position information of its pixels. Therefore, when processing images, the pose recognition model considers not only the influence of the image information itself on the pose of the target object but also the influence of the position information of the pixels in the target image on the pose of the target object. The factors considered are relatively comprehensive. Therefore, the pose of the target object obtained in this way has high accuracy, which is beneficial to improving the accuracy of the motion information of the acquired target image.

[0022] In one possible implementation, the location information includes the coordinates of pixels in an image coordinate system constructed based on the target image. In the aforementioned implementation, a vertex of the target image can be used as the origin of the image coordinate system. This allows determination of the coordinates of all pixels in the target image within the image coordinate system. This information can then be used as input to the pose recognition model. Thus, the input to the pose recognition model includes not only each pixel in the target image but also the coordinates of each pixel in the target image within the image coordinate system. By inputting this information from the target region into the pose recognition model, the model can effectively consider the influence of the location information of all pixels in the target region on the pose of the target object during image processing. Therefore, the pose of the target object obtained based on this information has high accuracy.

[0023] In one possible implementation, the pose recognition model is trained based on the predicted projection of the target object's predicted pose onto the image to be processed and the actual projection of the target object's true pose onto the image to be processed.

[0024] In one possible implementation, the pose of the target object includes the orientation of the target object in the camera coordinate system and the limb behavior of the target object in the camera coordinate system, which is constructed based on the camera that captures the target image.

[0025] In one possible implementation, the method further includes: processing the target image and the position information of the pixels in the target image through a pose recognition model to obtain shape parameters and displacement parameters, and using the pose parameters, shape parameters and displacement parameters together to obtain the pose of the target object.

[0026] In one possible implementation, the method further includes: normalizing the position information of pixels in the target image to obtain normalized position information; and processing the target image and the position information of pixels in the target image through a pose recognition model to obtain pose parameters, including: processing the target image and the normalized position information through a pose recognition model to obtain pose parameters.

[0027] In one possible implementation, the target object is the human body.

[0028] A third aspect of this application provides a model training method, which includes: acquiring an image to be processed; processing a target region of the image to be processed and the position information of the target region in the image to be processed using a model to be trained to obtain pose parameters, wherein the target region is the region where the target object is located; acquiring the predicted pose of the target object based on the pose parameters; and training the model to be trained based on the predicted pose and the true pose of the target object to obtain a pose recognition model.

[0029] The pose recognition model obtained by the above method can perceive the position of the target object in the entire image and learn the relationship between that position and the pose of the target object, thereby capturing the pose of the target object more accurately. When the pose recognition model is used in image processing, the input of the pose recognition model not only includes the cropped target region, but also the positional information of the target region in the target image. Therefore, the pose recognition model not only considers the influence of the image information of the target region itself on the pose of the target object, but also the influence of the positional information of the target region in the target image on the pose of the target object. The factors considered are more comprehensive. Therefore, the pose of the target object obtained based on this method has high accuracy, which is conducive to improving the accuracy of the motion information of the acquired target image.

[0030] In one possible implementation, the pose recognition model is trained based on the predicted pose and the true pose of the target object. This includes training the model based on the predicted projection of the target object's predicted pose onto the image to be processed and the true projection of the target object's true pose onto the image to be processed.

[0031] In one possible implementation, the location information includes the coordinates of the center point of the target region in the image coordinate system and the size of the target region, the image coordinate system being constructed based on the image to be processed.

[0032] In one possible implementation, the location information includes the coordinates of the vertices of the target region in an image coordinate system, which is constructed based on the image to be processed.

[0033] In one possible implementation, the predicted pose of the target object includes the predicted orientation of the target object in the camera coordinate system and the predicted limb behavior of the target object in the camera coordinate system, which is constructed based on the camera that captured the image to be processed.

[0034] In one possible implementation, the method includes: processing the target region of the image to be processed and the position information of the target region in the image to be processed through the model to be trained to obtain shape parameters and displacement parameters, and the pose parameters, shape parameters and displacement parameters are used together to obtain the pose of the target object.

[0035] In one possible implementation, the method further includes: normalizing the position information of the target region in the image to be processed to obtain normalized position information; and processing the target region of the image to be processed and the position information of the target region in the image to be processed through the model to be trained to obtain pose parameters, including: processing the target region of the image to be processed and the normalized position information through the model to be trained to obtain pose parameters.

[0036] In one possible implementation, the target object is the human body.

[0037] A fourth aspect of this application provides a model training method, which includes: acquiring an image to be processed; processing the image to be processed and the position information of pixels in the image to be processed using a model to be trained to obtain pose parameters; acquiring the predicted pose of a target object based on the pose parameters; and training the model to be trained based on the predicted pose and the true pose of the target object to obtain a pose recognition model.

[0038] The pose recognition model obtained by the above method can perceive the position of the target object in the entire image and learn the relationship between that position and the pose of the target object, thereby capturing the pose of the target object more accurately. When the pose recognition model is used for image processing, its input includes not only the target image but also the positional information of the pixels in the target image. Therefore, the pose recognition model considers not only the influence of the image information of the target image itself on the pose of the target object but also the influence of the positional information of the pixels in the target image on the pose of the target object. The factors considered are more comprehensive, so the pose of the target object obtained based on this method has high accuracy, which in turn helps to improve the accuracy of the motion information of the acquired target image.

[0039] In one possible implementation, the model to be trained is trained based on the predicted pose and the true pose of the target object to obtain a pose recognition model; the model to be trained is trained based on the predicted projection result of the predicted pose of the target object on the image to be processed and the true projection result of the true pose of the target object on the image to be processed to obtain a pose recognition model.

[0040] In one possible implementation, the location information includes the coordinates of the pixel in an image coordinate system constructed based on the image to be processed.

[0041] In one possible implementation, the predicted pose of the target object includes the predicted orientation of the target object in the camera coordinate system and the predicted limb behavior of the target object in the camera coordinate system, which is constructed based on the camera that captured the image to be processed.

[0042] In one possible implementation, the method further includes: processing the target region of the image to be processed and the position information of the pixels in the image to be processed through the model to be trained to obtain pose parameters, shape parameters and displacement parameters, which are used together to obtain the pose of the target object.

[0043] In one possible implementation, the method further includes: normalizing the position information of pixels in the image to be processed to obtain normalized position information; and processing the image to be processed and the position information of pixels in the image to be processed through the model to be trained to obtain pose parameters, including: processing the image to be processed and the normalized position information through the model to be trained to obtain pose parameters.

[0044] In one possible implementation, the target object is the human body.

[0045] A fifth aspect of this application provides a posture recognition device, which includes: an acquisition module for acquiring a target image; and a processing module for processing a target region of the target image and the position information of the target region in the target image through a posture recognition model to obtain posture parameters, wherein the target region is the region where the target object is located, and the posture parameters are used to obtain the posture of the target object.

[0046] As can be seen from the above device, after acquiring the target image, the target region where the target object is located and the position information of the target region in the target image can be input into the pose recognition model. The pose recognition model processes the target region and its position information in the target image to obtain pose parameters. Therefore, the pose of the target object can be obtained based on the pose parameters. In the aforementioned process, the input to the pose recognition model includes not only the cropped target region but also the position information of the target region in the target image. Therefore, when performing image processing, the pose recognition model considers not only the influence of the image information of the target region itself on the pose of the target object but also the influence of the position information of the target region in the target image on the pose of the target object. The factors considered are relatively comprehensive. Therefore, the pose of the target object obtained in this way has high accuracy, which helps to improve the accuracy of the motion information of the acquired target image.

[0047] In one possible implementation, the location information includes the coordinates of the center point of the target region in the image coordinate system and the size of the target region, the image coordinate system being constructed based on the target image.

[0048] In one possible implementation, the location information includes the coordinates of the vertices of the target region in an image coordinate system, which is constructed based on the target image.

[0049] In one possible implementation, the pose recognition model is trained based on the predicted projection of the target object's predicted pose onto the image to be processed and the true projection of the target object's true pose onto the image to be processed.

[0050] In one possible implementation, the pose of the target object includes the orientation of the target object in the camera coordinate system and the limb behavior of the target object in the camera coordinate system, which is constructed based on the camera that captures the target image.

[0051] In one possible implementation, the processing module is further configured to process the target region of the target image and the position information of the target region in the target image through a pose recognition model to obtain shape parameters and displacement parameters. The pose parameters, shape parameters, and displacement parameters are used together to obtain the pose of the target object.

[0052] In one possible implementation, the device further includes: a normalization module for normalizing the position information of the target region in the target image to obtain normalized position information; and a processing module for processing the target region of the target image and the normalized position information through a pose recognition model to obtain pose parameters.

[0053] In one possible implementation, the target object is the human body.

[0054] A sixth aspect of this application provides a pose recognition device, which includes: an acquisition module for acquiring a target image; and a processing module for processing the target image and the position information of pixels in the target image through a pose recognition model to obtain pose parameters, wherein the pose parameters are used to acquire the pose of a target object contained in the target image.

[0055] As can be seen from the above device, after acquiring the target image, the target image and the position information of its pixels can be input into the pose recognition model. The pose recognition model processes the target image and the position information of its pixels to obtain pose parameters. Therefore, the pose of the target object presented in the target image can be obtained based on the pose parameters. In the aforementioned process, the input to the pose recognition model includes not only the target image but also the position information of its pixels. Therefore, when performing image processing, the pose recognition model considers not only the influence of the image information of the target image itself on the pose of the target object but also the influence of the position information of the pixels in the target image on the pose of the target object. The factors considered are relatively comprehensive. Therefore, the pose of the target object obtained in this way has high accuracy, which is beneficial to improving the accuracy of the motion information of the acquired target image.

[0056] In one possible implementation, the location information includes the coordinates of the pixel in an image coordinate system constructed based on the target image.

[0057] In one possible implementation, the pose recognition model is trained based on the predicted projection of the target object's predicted pose onto the image to be processed and the true projection of the target object's true pose onto the image to be processed.

[0058] In one possible implementation, the pose of the target object includes the orientation of the target object in the camera coordinate system and the limb behavior of the target object in the camera coordinate system, which is constructed based on the camera that captures the target image.

[0059] In one possible implementation, the processing module is further configured to process the target image and the position information of the pixels in the target image through a pose recognition model to obtain shape parameters and displacement parameters. The pose parameters, shape parameters, and displacement parameters are used together to obtain the pose of the target object.

[0060] In one possible implementation, the device further includes: a normalization module for normalizing the position information of pixels in the target image to obtain normalized position information; and a processing module for processing the target image and the normalized position information through a pose recognition model to obtain pose parameters.

[0061] In one possible implementation, the target object is the human body.

[0062] A seventh aspect of this application provides a model training apparatus, comprising: a first acquisition module for acquiring an image to be processed; a processing module for processing a target region of the image to be processed and the position information of the target region in the image to be processed using a model to be trained, thereby obtaining pose parameters, wherein the target region is the region where a target object is located; a second acquisition module for acquiring a predicted pose of the target object based on the pose parameters; and a training module for training the model to be trained based on the predicted pose and the true pose of the target object, thereby obtaining a pose recognition model.

[0063] The pose recognition model obtained by the aforementioned device can perceive the position of the target object in the entire image and learn the relationship between that position and the pose of the target object, thereby capturing the pose of the target object more accurately. When the pose recognition model is used for image processing, the input to the pose recognition model includes not only the cropped target region but also the positional information of the target region in the target image. Therefore, the pose recognition model considers not only the influence of the image information of the target region itself on the pose of the target object but also the influence of the positional information of the target region in the target image on the pose of the target object. The factors considered are more comprehensive, so the pose of the target object obtained based on this method has high accuracy, which in turn helps to improve the accuracy of the motion information of the acquired target image.

[0064] In one possible implementation, a training module is used to train the training model based on the predicted projection of the target object's predicted pose onto the image to be processed and the true projection of the target object's true pose onto the image to be processed, thereby obtaining a pose recognition model.

[0065] In one possible implementation, the location information includes the coordinates of the center point of the target region in the image coordinate system and the size of the target region, the image coordinate system being constructed based on the image to be processed.

[0066] In one possible implementation, the location information includes the coordinates of the vertices of the target region in an image coordinate system, which is constructed based on the image to be processed.

[0067] In one possible implementation, the predicted pose of the target object includes the predicted orientation of the target object in the camera coordinate system and the predicted limb behavior of the target object in the camera coordinate system, which is constructed based on the camera that captured the image to be processed.

[0068] In one possible implementation, the processing module is further configured to process the target region of the image to be processed and the position information of the target region in the image to be processed through the pose recognition model to obtain shape parameters and displacement parameters. The pose parameters, shape parameters and displacement parameters are used together to obtain the pose of the target object.

[0069] In one possible implementation, the device further includes: a normalization module for normalizing the position information of the target region in the image to be processed to obtain normalized position information; and a processing module for processing the target region of the image to be processed and the normalized position information through the model to be trained to obtain pose parameters.

[0070] In one possible implementation, the target object is the human body.

[0071] An eighth aspect of this application provides a model training apparatus, comprising: a first acquisition module for acquiring an image to be processed; a processing module for processing the image to be processed and the position information of pixels in the image to be processed using a model to be trained, thereby obtaining pose parameters; a second acquisition module for acquiring a predicted pose of a target object based on the pose parameters; and a training module for training the model to be trained based on the predicted pose and the true pose of the target object, thereby obtaining a pose recognition model.

[0072] The pose recognition model obtained by the aforementioned device can perceive the position of the target object in the entire image and learn the relationship between that position and the pose of the target object, thereby capturing the pose of the target object more accurately. When the pose recognition model is processing images, its input includes not only the target image but also the positional information of the pixels in the target image. Therefore, the pose recognition model considers not only the influence of the image information of the target image itself on the pose of the target object but also the influence of the positional information of the pixels in the target image on the pose of the target object. The factors considered are relatively comprehensive, so the pose of the target object obtained based on this method has high accuracy, which in turn helps to improve the accuracy of the motion information of the acquired target image.

[0073] In one possible implementation, a training module is used to train the training model based on the predicted projection of the target object's predicted pose onto the image to be processed and the true projection of the target object's true pose onto the image to be processed, thereby obtaining a pose recognition model.

[0074] In one possible implementation, the location information includes the coordinates of the pixel in an image coordinate system constructed based on the image to be processed.

[0075] In one possible implementation, the predicted pose of the target object includes the predicted orientation of the target object in the camera coordinate system and the predicted limb behavior of the target object in the camera coordinate system, which is constructed based on the camera that captured the image to be processed.

[0076] In one possible implementation, the processing module is further configured to process the target region of the image to be processed and the position information of the pixels in the image to be processed through the pose recognition model to obtain shape parameters and displacement parameters. The pose parameters, shape parameters and displacement parameters are used together to obtain the pose of the target object.

[0077] In one possible implementation, the device further includes: a normalization module for normalizing the position information of pixels in the image to be processed to obtain normalized position information; and a processing module for processing the image to be processed and the normalized position information using the model to be trained to obtain pose parameters.

[0078] In one possible implementation, the target object is the human body.

[0079] A ninth aspect of this application provides a posture recognition device, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the posture recognition device performs the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0080] A tenth aspect of this application provides a model training apparatus, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the model training apparatus performs the method described in the third aspect, any possible implementation of the third aspect, the fourth aspect, or any possible implementation of the fourth aspect.

[0081] The eleventh aspect of this application provides a circuit system including a processing circuit configured to perform the method described in the first aspect, any possible implementation of the first aspect, the second aspect, any possible implementation of the second aspect, the third aspect, any possible implementation of the third aspect, the fourth aspect, or any possible implementation of the fourth aspect.

[0082] The twelfth aspect of this application provides a chip system including a processor for calling a computer program or computer instructions stored in a memory, such that the processor executes the method described in the first aspect, any possible implementation of the first aspect, the second aspect, any possible implementation of the second aspect, the third aspect, any possible implementation of the third aspect, the fourth aspect, or any possible implementation of the fourth aspect.

[0083] In one possible implementation, the processor is coupled to the memory via an interface.

[0084] In one possible implementation, the chip system also includes a memory that stores computer programs or computer instructions.

[0085] The thirteenth aspect of this application provides a computer storage medium storing a computer program that, when executed by a computer, causes the computer to perform the method as described in the first aspect, any possible implementation of the first aspect, the second aspect, any possible implementation of the second aspect, the third aspect, any possible implementation of the third aspect, the fourth aspect, or any possible implementation of the fourth aspect.

[0086] The fourteenth aspect of this application provides a computer program product storing instructions that, when executed by a computer, cause the computer to perform the method as described in the first aspect, any possible implementation of the first aspect, the second aspect, any possible implementation of the second aspect, the third aspect, any possible implementation of the third aspect, the fourth aspect, or any possible implementation of the fourth aspect.

[0087] In this embodiment, after acquiring the target image, the target region where the target object is located and its position information within the target image can be input into the pose recognition model. The pose recognition model processes the target region and its position information to obtain pose parameters. Therefore, the pose of the target object can be obtained based on these pose parameters. In the aforementioned process, the input to the pose recognition model includes not only the cropped target region but also its position information within the target image. Thus, when processing the image, the pose recognition model considers not only the influence of the target region's image information on the target object's pose but also the influence of the target region's position information within the target image on the target object's pose. This comprehensive consideration results in a more accurate target object pose, thereby improving the accuracy of the motion information obtained from the target image. Attached Figure Description

[0088] Figure 1 This is a schematic diagram of the related technology;

[0089] Figure 2 This is another schematic diagram of the related technology;

[0090] Figure 3 A structural diagram illustrating the main framework of artificial intelligence;

[0091] Figure 4a A schematic diagram of the structure of an image processing system provided in an embodiment of this application;

[0092] Figure 4b This is another schematic diagram of the image processing system provided in the embodiments of this application;

[0093] Figure 4c A schematic diagram of an image processing device provided in an embodiment of this application;

[0094] Figure 5 A schematic diagram of the system 100 architecture provided in the embodiments of this application;

[0095] Figure 6 A flowchart illustrating the pose recognition method provided in this application embodiment;

[0096] Figure 7 This is a schematic diagram illustrating an application example of pose recognition provided in an embodiment of this application;

[0097] Figure 8 This is another schematic diagram of the pose recognition method provided in the embodiments of this application;

[0098] Figure 9a This is a schematic diagram illustrating another application example of pose recognition provided in the embodiments of this application;

[0099] Figure 9b A schematic diagram of the structure of the pose recognition model provided in the embodiments of this application;

[0100] Figure 10 A schematic diagram of the model training method provided in the embodiments of this application;

[0101] Figure 11 A schematic diagram of the projection result provided in an embodiment of this application;

[0102] Figure 12 Another schematic diagram of the model training method provided in the embodiments of this application;

[0103] Figure 13 A schematic diagram of the posture recognition device provided in the embodiments of this application;

[0104] Figure 14 This is another structural schematic diagram of the posture recognition device provided in the embodiments of this application;

[0105] Figure 15 Another schematic diagram of the model training device provided in the embodiments of this application;

[0106] Figure 16 Another schematic diagram of the model training device provided in the embodiments of this application;

[0107] Figure 17 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0108] Figure 18 A schematic diagram of the structure of the training device provided in the embodiments of this application;

[0109] Figure 19 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0110] This application provides a posture recognition method and related device, which obtains the posture of the target object after processing an image containing the target object with high accuracy, thereby improving the accuracy of the motion information of the acquired target image.

[0111] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0112] Human motion capture technology is commonly used in the film, television, and gaming industries. This technology processes input video streams to capture the posture of the human body in each frame, thereby obtaining the body's motion information. Based on this technology, posture data of human movement can be extracted from massive amounts of video, and it has a wide range of applications, such as virtual character driving in augmented reality (AR) and virtual reality (VR) scenarios, remote conferencing, and the metaverse, etc.

[0113] Currently, human motion capture can be achieved using AI technology. Specifically, for a given image, a trained neural network can process the region containing the human body in the image to obtain pose parameters, which can then be used to determine the human body's posture.

[0114] like Figure 1 As shown ( Figure 1 (This is a schematic diagram of a related technology.) In this technology, if there are multiple images presenting the same environment, namely Image 1, Image 2, and Image 3, and a human body is located in three different positions within that environment in these three images, and the human body's posture is extremely similar regardless of the position, the neural network will determine that the posture parameters of the human body in these three images are consistent; that is, it will determine that posture 1 of the human body in Image 1, posture 2 of the human body in Image 2, and posture 3 of the human body in Image 3 are the same. However, in a real environment, once the human body moves, although the posture of the human body in different positions is similar, there will definitely be subtle differences between the postures, such as... Figure 2 As shown ( Figure 2 (This is another illustration of the related technology). Observing the human body in three positions in the environment from a top-down perspective, for posture 1, the human body is facing sideways (tilted to the right) towards the camera; for posture 2, the human body is facing directly towards the camera; and for posture 3, the human body is facing sideways (tilted to the left) towards the camera. It is evident that the differences between these three postures are not recognized by the neural network in the related technology, resulting in inaccurate capture of the human body posture and consequently affecting the accuracy of the human motion information.

[0115] To address the aforementioned problems, this application provides a posture recognition method that can be implemented in conjunction with artificial intelligence (AI) technology. AI technology is a discipline that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence. AI technology achieves optimal results by perceiving the environment, acquiring knowledge, and using that knowledge. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. Using artificial intelligence for image processing is a common application of AI.

[0116] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 3 , Figure 3This is a structural diagram illustrating the main framework of artificial intelligence. The following explanation of the AI ​​framework is based on two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

[0117] (1) Infrastructure

[0118] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0119] (2) Data

[0120] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0121] (3) Data processing

[0122] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0123] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0124] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0125] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0126] (4) General ability

[0127] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0128] (5) Smart Products and Industry Applications

[0129] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0130] The following section introduces several application scenarios for this application.

[0131] Figure 4a This is a schematic diagram of an image processing system provided in an embodiment of this application. The image processing system includes a user device and a data processing device. The user device includes smart terminals such as mobile phones, personal computers, or information processing centers. The user device is the initiator of image processing; as the initiator of image processing requests, requests are typically initiated by the user through the user device.

[0132] The aforementioned data processing equipment can be devices or servers with data processing capabilities, such as cloud servers, network servers, application servers, and management servers. The data processing equipment receives image processing requests from smart terminals through an interactive interface, and then performs image processing methods such as machine learning, deep learning, search, reasoning, and decision-making through a storage device and a data processing processor. The storage device in the data processing equipment can be a general term, including local storage and a database storing historical data. The database can be located on the data processing equipment or on other network servers.

[0133] exist Figure 4a In the image processing system shown, the user equipment can receive user instructions. For example, the user equipment can acquire an image input / selected by the user and then send a request to the data processing device, causing the data processing device to perform image processing applications (e.g., target object pose recognition, etc.) on the image acquired by the user equipment, thereby obtaining the corresponding processing result for the image. For instance, the user equipment can acquire an image input by the user and then send a target object pose recognition request to the data processing device, causing the data processing device to classify the image, thereby obtaining the pose parameters of the target object in the image, and thus determining the pose of the target object in the image.

[0134] exist Figure 4aIn this context, the data processing device can execute the image processing method of the embodiments of this application.

[0135] Figure 4b This is another schematic diagram of the image processing system provided in the embodiments of this application. Figure 4b In this context, the user equipment (UE) directly functions as a data processing device. This UE can directly acquire input from the user and process it directly through its own hardware. The specific process is similar to... Figure 4a Similar to the description above, it will not be repeated here.

[0136] exist Figure 4b In the image processing system shown, the user equipment can receive instructions from the user. For example, the user equipment can acquire an image selected by the user and then perform image processing applications (such as the recognition of the pose of the target object) on the image to obtain the corresponding processing result for the image.

[0137] exist Figure 4b In this context, the user equipment itself can execute the posture recognition method of the embodiments of this application.

[0138] Figure 4c This is a schematic diagram of an image processing device provided in an embodiment of this application.

[0139] The above Figure 4a and Figure 4b The user equipment in the context can specifically be Figure 4c Local device 301 or local device 302 in the system. Figure 4a The data processing equipment in the middle can specifically be Figure 4c The execution device 210 in the process includes a data storage system 250 that can store the data to be processed by the execution device 210. The data storage system 250 can be integrated into the execution device 210 or set up in the cloud or on other network servers.

[0140] Figure 4a and Figure 4b The processor in the image can be trained on data using neural network models or other models (e.g., support vector machine-based models) for machine learning / deep learning, and then use the trained or learned models to perform image processing applications on the image to obtain the corresponding processing results.

[0141] Figure 5 A schematic diagram of the system 100 architecture provided in this application embodiment, in Figure 5In the process, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. The input data in this embodiment may include various scheduled tasks, callable resources, and other parameters.

[0142] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processing (such as implementing the neural network function in this application), the execution device 110 may call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0143] Finally, I / O interface 112 returns the processing result to client device 140, thereby providing it to the user.

[0144] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different objectives or tasks. These target models / rules can then be used to achieve the aforementioned objectives or complete the aforementioned tasks, thereby providing the user with the required results. The training data can be stored in the database 130 and originates from training samples collected by the data acquisition device 160.

[0145] exist Figure 5 In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0146] It is worth noting that, Figure 5 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 5 In this context, the data storage system 150 is an external memory relative to the execution device 110. However, in other cases, the data storage system 150 can also be placed within the execution device 110. For example... Figure 5 As shown, a neural network can be trained using training device 120.

[0147] This application also provides a chip including a neural network processor (NPU). This chip can be configured as follows: Figure 5 The execution device 110 shown is used to perform the calculations of the calculation module 111. This chip can also be located in, for example... Figure 5 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rules.

[0148] The Neural Processing Unit (NPU) is a coprocessor mounted on the main central processing unit (CPU) (host CPU), where tasks are assigned by the CPU. The core of the NPU is the computation circuitry, which is controlled by a controller to retrieve data from memory (weight memory or input memory) and perform calculations.

[0149] In some implementations, the arithmetic circuitry includes multiple process engines (PEs). In some implementations, the arithmetic circuitry is a two-dimensional pulsating array. The arithmetic circuitry can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuitry is a general-purpose matrix processor.

[0150] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory and caches it in each PE (Process Equipment) of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory and performs matrix operations with matrix B. The partial or final result of the obtained matrix is ​​stored in the accumulator.

[0151] Vector computation units can further process the output of computational circuits, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, vector computation units can be used for computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, and local response normalization.

[0152] In some implementations, the vector computation unit can store the processed output vector into a unified buffer. For example, the vector computation unit can apply a nonlinear function to the output of the arithmetic circuit, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit, for example, for use in subsequent layers of a neural network.

[0153] The unified memory is used to store input data and output data.

[0154] The weight data is directly transferred from the external memory to the input memory and / or unified memory, stored in the weight memory, and stored in the unified memory to the external memory through the direct memory access controller (DMAC).

[0155] The bus interface unit (BIU) is used to enable interaction between the main CPU, DMAC, and instruction fetch memory via a bus.

[0156] The instruction fetch buffer, connected to the controller, is used to store the instructions used by the controller.

[0157] The controller is used to invoke instructions cached in the memory to control the operation of the computing accelerator.

[0158] Generally, the unified memory, input memory, weight memory, and instruction fetch memory are all on-chip memories, while external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDRSDRAM), high bandwidth memory (HBM), or other readable and writable memories.

[0159] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0160] (1) Neural Network

[0161] A neural network can be composed of neural units, which can be operational units that take xs and an intercept of 1 as inputs, and whose output can be:

[0162]

[0163] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0164] The work of each layer in a neural network can be described by the mathematical expression y = a(Wx + b). From a physical perspective, the work of each layer in a neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space. These five operations include: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are performed by Wx, operation 4 by +b, and operation 5 by a(). The term "space" is used here because the objects being classified are not individual things, but a class of things, and space refers to the set of all individuals of this class of things. Here, W is the weight vector, and each value in this vector represents the weight value of a neuron in that layer of the neural network. This vector W determines the spatial transformation from the input space to the output space mentioned above; that is, the weights W of each layer control how the space is transformed. The purpose of training a neural network is to ultimately obtain the weight matrix of all layers of the trained neural network (a weight matrix formed by the vectors W of many layers). Therefore, the training process of a neural network is essentially about learning how to control the transformation space, and more specifically, learning the weight matrix.

[0165] Because we want the output of the neural network to be as close as possible to the actual predicted value, we can compare the current network's prediction with the desired target value, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring the parameters of each layer in the neural network). For example, if the network's prediction is too high, the weight vector is adjusted to make it predict lower, and this adjustment is continued until the neural network can predict the actual target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value," which is the loss function or objective function. These are important equations used to measure the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so training the neural network becomes the process of minimizing this loss as much as possible.

[0166] (2) Backpropagation algorithm

[0167] Neural networks can employ backpropagation (BP) to correct the parameters of the initial neural network model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the initial neural network model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0168] The method provided in this application is described below from the perspectives of neural network training and neural network application.

[0169] The model training method provided in this application involves image processing and can be specifically applied to data processing methods such as data training, machine learning, and deep learning. It performs symbolic and formal intelligent information modeling, extraction, preprocessing, and training on training data (such as the image to be processed in this application) to ultimately obtain a trained neural network (such as the pose recognition model in this application). Furthermore, the pose recognition method provided in this application can utilize the trained neural network to input input data (such as the target image and target region of the target image in this application) into the trained neural network to obtain output data (such as pose parameters in this application). It should be noted that the model training method and pose recognition method provided in this application are inventions based on the same concept and can be understood as two parts of a system or two stages of an overall process: such as the model training stage and the model application stage.

[0170] Figure 6 A flowchart illustrating the pose recognition method provided in this application embodiment is shown below. Figure 6 As shown, the method includes:

[0171] 601. Obtain the target image.

[0172] In this embodiment, when pose recognition of a target image is required, the target image can be acquired first. The target image can be a frame from a video stream or a single image. The target image typically presents the target object and its environment. Therefore, the goal of pose recognition is to obtain the pose of the target object within the target image. For example, ... Figure 7 As shown ( Figure 7 (This is a schematic diagram of an application example of posture recognition provided in the embodiments of this application). The content presented in the target image is the human body and the environment in which the human body is located. Therefore, the goal of posture recognition of the target image is to identify the posture of the human body presented in the target image.

[0173] To reduce the computational burden of pose recognition models, target images can be preprocessed. Specifically, the target image can be detected to determine the region where the target object is located; this region can be called the target region (or detection box). Continuing with the example above, after detecting the region where the human body is located in the target image, this region can be defined as the target region (i.e., the detection box). Figure 7 (The area enclosed in red in the middle).

[0174] Furthermore, the positional information of the target region in the target image can be obtained. This positional information can exist in several ways: (1) The positional information of the target region in the target image includes the coordinates of the center point of the target region in the image coordinate system and the size of the target region. The image coordinate system is constructed based on the target image. Specifically, a vertex of the target image (e.g., the vertex at the top left corner of the target image) can be used as the origin of the image coordinate system. Then, the coordinates of the center point of the target region in the image coordinate system, as well as the length and width of the target region (which is equivalent to obtaining the size of the target region), can be determined. This information can be used to indicate the position of the target region in the target image. (2) The positional information of the target region in the target image includes the coordinates of the vertices of the target region in the image coordinate system. The image coordinate system is constructed based on the target image. Specifically, a vertex of the target image (e.g., the vertex at the top left corner of the target image) can be used as the origin of the image coordinate system. Then, the coordinates of all vertices of the target region in the image coordinate system can be determined. This information can be used to indicate the position of the target region in the target image, etc.

[0175] Furthermore, after obtaining the location information of the target region in the target image, this location information can be normalized to obtain the normalized location information of the target region in the target image. Specifically, let the location information of the target region in the target image be I, and the location information of the normalized target region in the target image be I'. Then, I' = (I - mean(I)) / F, F = sqrt(w×w + h×h), where mean(I) is the mean of the target region, w is the width of the target region, and h is the length of the target region. Obtaining the normalized information in this way is beneficial for subsequent pose recognition operations of the model.

[0176] 602. The target region of the target image and the position information of the target region in the target image are processed by the pose recognition model to obtain the pose parameters. The target region is the region where the target object is located, and the pose parameters are used to obtain the pose of the target object.

[0177] After obtaining the target region and its normalized location information within the target image, these information can be input into a pose recognition model (a trained neural network). The model then processes this information (e.g., feature extraction) to obtain pose parameters. Based on these pose parameters, the pose of the target object in the target image can then be determined.

[0178] It should be noted that the pose recognition model can output not only pose parameters, but also shape and displacement parameters. The pose parameters typically consist of two parts: one part indicates the rotation angle of the target object relative to the camera, and the other part indicates the angles between the joints of the target object. Shape parameters indicate the three-dimensional shape of the target object. Displacement parameters indicate the extent of the target object's position within the target region and its offset within the target region (e.g., the degree to the left or right of the target object relative to the center point of the target region). Based on the pose, shape, and displacement parameters, the pose of the target object can be calculated. The pose of the target object can be represented by multiple three-dimensional keypoints (3D keypoints). These 3D keypoints describe the target object's orientation in the camera coordinate system (i.e., the target object's orientation relative to the camera) and its limb behavior in the camera coordinate system (i.e., the target object's movement in three-dimensional space). The origin of this camera coordinate system is the camera that captured the target image. As in the example above... Figure 7As shown, after the posture recognition model outputs the posture parameters θ, shape parameters β, and displacement parameters S and T, these parameters can be calculated to obtain multiple 3D key points of the target object. These 3D key points can be combined to present the orientation of the target object in the camera coordinate system and the limb behavior of the target object in the camera coordinate system.

[0179] This completes the pose recognition of the target image. If the target image is a single frame in a video stream, the same operations can be performed on the remaining frames in the video stream to obtain multiple consecutive poses of the target object, forming the motion information of the target object and thus meeting various application needs of users.

[0180] In this embodiment, after acquiring the target image, the target region where the target object is located and its position information within the target image can be input into the pose recognition model. The pose recognition model processes the target region and its position information to obtain pose parameters. Therefore, the pose of the target object can be obtained based on these pose parameters. In the aforementioned process, the input to the pose recognition model includes not only the cropped target region but also its position information within the target image. Thus, when processing the image, the pose recognition model considers not only the influence of the target region's image information on the target object's pose but also the influence of the target region's position information within the target image on the target object's pose. This comprehensive consideration results in a more accurate target object pose, thereby improving the accuracy of the motion information obtained from the target image.

[0181] Figure 8 This is another flowchart illustrating the pose recognition method provided in the embodiments of this application, as shown below. Figure 8 As shown, the method includes:

[0182] 801. Obtain the target image.

[0183] In this embodiment, when pose recognition of a target image is required, the target image can be acquired first. The target image can be a frame from a video stream or a single image. The target image typically presents the target object and its environment. Therefore, the goal of pose recognition is to obtain the pose of the target object within the target image. For example, ... Figure 9a As shown ( Figure 9a (This is a schematic diagram of another application example of posture recognition provided in the embodiments of this application). The content presented in the target image is the human body and the environment in which the human body is located. Therefore, the goal of posture recognition of the target image is to identify the posture of the human body presented in the target image.

[0184] Furthermore, the positional information of pixels in the target image can be obtained. This positional information includes the coordinates of the pixels in the target image within the image coordinate system, which is constructed based on the target image. Specifically, a vertex of the target image (e.g., the vertex at the top left corner of the target image) can be used as the origin of the image coordinate system. Then, the coordinates of all pixels in the target image within the image coordinate system can be determined. This information can be used as input to the pose recognition model. In this way, the input to the pose recognition model includes not only each pixel in the target image but also the coordinates of each pixel in the target image within the image coordinate system.

[0185] Furthermore, after obtaining the positional information of pixels in the target image, this positional information can be normalized to obtain the normalized positional information of pixels in the target image. Specifically, let the positional information of pixels in the target image be I, and the positional information of pixels in the normalized target image be I'. Then, I' = (I - mean(I)) / F, F = sqrt(w × w + h × h), where mean(I) is the mean of the target image, w is the width of the target image, and h is the length of the target image. Obtaining the normalized information in this way is beneficial for subsequent pose recognition operations of the model.

[0186] 802. The pose recognition model is used to process the target image and the position information of the pixels in the target image to obtain pose parameters. The pose parameters are used to obtain the pose of the target object contained in the target image.

[0187] After obtaining the target image and the positional information of the pixels in the normalized target image, these information can be input into a pose recognition model (a trained neural network). The pose recognition model then processes the target region of the target image and the positional information of the pixels in the normalized target image (e.g., feature extraction and other operations) to obtain pose parameters. Based on these pose parameters, the pose of the target object in the target image can be obtained.

[0188] like Figure 9b As shown ( Figure 9bThis is a schematic diagram of the pose recognition model provided in an embodiment of this application. The pose recognition model can include two parts: an encoder and multiple convolutional layers. After receiving the target image, the encoder performs feature extraction processing on the target image to obtain a feature map, which is then sent to the convolutional layers. The multiple convolutional layers perform at least one convolution process on the feature map and the location information of points in the normalized target image to obtain pose parameters. Therefore, the pose of the target object in the target image can be determined based on these pose parameters.

[0189] In one possible implementation, the method further includes: processing the target image and the position information of the pixels in the target image through a pose recognition model to obtain shape parameters and displacement parameters, and using the pose parameters, shape parameters and displacement parameters together to obtain the pose of the target object.

[0190] In one possible implementation, the pose of the target object includes the orientation of the target object in the camera coordinate system and the limb behavior of the target object in the camera coordinate system, which is constructed based on the camera that captures the target image.

[0191] For instructions on step 802, please refer to [link / reference]. Figure 6 The relevant descriptions of step 602 in the illustrated embodiment will not be repeated here.

[0192] It should be understood that this embodiment is only used as an example. Figure 9b The examples shown are for illustrative purposes only and do not limit the structure of the pose recognition model in this application. The pose recognition model can also have various other structures, as long as it can realize the pose recognition function.

[0193] In this embodiment, after acquiring the target image, the target image and the position information of its pixels can be input into a pose recognition model. The pose recognition model processes the target image and the position information of its pixels to obtain pose parameters. Therefore, the pose of the target object presented in the target image can be obtained based on the pose parameters. In the aforementioned process, the input to the pose recognition model includes not only the target image but also the position information of its pixels. Therefore, when processing the image, the pose recognition model considers not only the influence of the image information of the target image itself on the pose of the target object but also the influence of the position information of the pixels in the target image on the pose of the target object. The factors considered are comprehensive, so the pose of the target object obtained in this way has high accuracy, which helps to improve the accuracy of the motion information of the acquired target image.

[0194] Furthermore, the pose recognition model provided in this application embodiment can be compared with the pose recognition model of related technologies, and the comparison results are shown in Tables 1 and 2:

[0195] Table 1

[0196]

[0197]

[0198] Table 2

[0199]

[0200] As shown in Table 1, on Dataset 1, compared with other related technologies, the posture recognition model provided by this application embodiment has significantly reduced all error indicators, that is, the posture recognition model provided by this application embodiment has superior performance in all error indicators.

[0201] As shown in Table 2, on both dataset 1 and dataset 2, compared to other related technologies, the posture recognition model provided by this application embodiment has significantly reduced all error indicators, meaning that the posture recognition model provided by this application embodiment also has superior performance in all error indicators.

[0202] The above is a detailed description of the pose recognition method provided in the embodiments of this application. The model training method provided in the embodiments of this application will be introduced below. Figure 10 A schematic diagram of the model training method provided in the embodiments of this application, as shown below. Figure 10 As shown, the method includes:

[0203] 1001. Obtain the image to be processed.

[0204] When training a model is required, a batch of training samples, i.e., images to be processed, can be obtained. It is worth noting that the true pose of the target object in the image to be processed is known.

[0205] After obtaining the image to be processed, detection processing can be performed on the image to determine the target region in the image. The location information of the target region in the image can also be obtained and normalized.

[0206] In one possible implementation, the target object is the human body.

[0207] In one possible implementation, the location information of the target region in the image to be processed includes the coordinates of the center point of the target region in the image coordinate system and the size of the target region, the image coordinate system being constructed based on the image to be processed.

[0208] In one possible implementation, the location information of the target region in the image to be processed includes the coordinates of the vertices of the target region in the image coordinate system, which is constructed based on the image to be processed.

[0209] For instructions on step 1001, please refer to [link / reference]. Figure 6 The relevant descriptions of step 601 in the illustrated embodiment will not be repeated here.

[0210] 1002. The target region of the image to be processed and the position information of the target region in the image to be processed are processed by the model to be trained to obtain the pose parameters. The target region is the region where the target object is located.

[0211] After obtaining the positional information of the image to be processed and the normalized target region in the image to be processed, the positional information of the image to be processed and the normalized target region in the image to be processed can be input into the model to be trained, so that the model to be trained can process the positional information of the target region in the image to be processed and the normalized target region in the image to be processed to obtain the pose parameters.

[0212] In one possible implementation, the method further includes: processing the target region of the image to be processed and the position information of the normalized target region in the image to be processed through a pose recognition model to obtain shape parameters and displacement parameters, and the pose parameters, shape parameters and displacement parameters are used together to obtain the pose of the target object.

[0213] For instructions on step 1002, please refer to [link / reference]. Figure 6 The relevant descriptions of step 602 in the illustrated embodiment will not be repeated here.

[0214] 1003. Based on the attitude parameters, obtain the predicted attitude of the target object.

[0215] After obtaining the attitude parameters, shape parameters, and displacement parameters, calculations can be performed based on these parameters to obtain the predicted attitude of the target object. It should be noted that the predicted attitude of the target object can be represented by multiple predicted 3D key points.

[0216] In one possible implementation, the predicted pose of the target object includes the predicted orientation of the target object in the camera coordinate system and the predicted limb behavior of the target object in the camera coordinate system, which is constructed based on the camera that captured the image to be processed.

[0217] For instructions on step 1003, please refer to [link / reference]. Figure 6 The relevant descriptions of step 602 in the illustrated embodiment will not be repeated here.

[0218] 1004. Based on the predicted pose and the true pose of the target object, train the model to be trained to obtain the pose recognition model.

[0219] After obtaining the predicted pose of the target object, since the true pose of the target object is known, the model to be trained can be trained based on the predicted pose and the true pose of the target object to obtain the pose recognition model.

[0220] In one possible implementation, training the model to be trained based on the predicted pose and the true pose of the target object to obtain a pose recognition model includes: training the model to be trained based on the predicted projection result of the target object's predicted pose on the image to be processed and the true projection result of the target object's true pose on the image to be processed to obtain a pose recognition model. Specifically, such as Figure 11 As shown ( Figure 11 (This is a schematic diagram of the projection result provided in an embodiment of this application). Since the predicted pose of the target object can be represented by multiple predicted 3D keypoints of the target object, and the predicted pose of the target object can be represented by multiple real 3D keypoints of the target object (these real 3D keypoints can be pre-marked and are known), the multiple predicted 3D keypoints of the target object can be projected onto the image to be processed to obtain multiple predicted 2D keypoints of the target object (i.e., the predicted projection result of the predicted pose of the target object on the image to be processed). Similarly, the multiple real 3D keypoints of the target object can also be projected onto the image to be processed to obtain multiple real 2D keypoints of the target object (i.e., the real projection result of the real pose of the target object on the image to be processed). Then, based on the multiple predicted 2D keypoints and multiple real 2D keypoints of the target object, the target loss can be calculated. The target loss is used to indicate the difference between the multiple predicted 2D keypoints and multiple real 2D keypoints of the target object. Then, the model parameters of the model to be trained can be updated based on the target loss, and the updated model can be trained using the next batch of training samples (i.e., steps 1002 to 1004 are re-executed) until the model training conditions are met (e.g., the target loss converges, etc.), thus obtaining... Figure 6 The pose recognition model in the illustrated embodiment.

[0221] The pose recognition model trained in this embodiment can perceive the position of a target object in the entire image and learn the relationship between that position and the target object's pose, thereby capturing the target object's pose more accurately. When the pose recognition model is used for image processing, its input includes not only the cropped target region but also the positional information of the target region within the target image. Therefore, the pose recognition model considers not only the influence of the target region's image information on the target object's pose but also the influence of the target region's positional information within the target image on the target object's pose. This comprehensive consideration results in higher accuracy of the target object's pose obtained through this method, thus improving the accuracy of the motion information obtained from the target image.

[0222] Figure 12 Another schematic diagram of the model training method provided in the embodiments of this application is shown below. Figure 12 As shown, the method includes:

[0223] 1201. Obtain the image to be processed.

[0224] When training a model is required, a batch of training samples, i.e., images to be processed, can be obtained. It is worth noting that the true pose of the target object in the image to be processed is known.

[0225] After obtaining the image to be processed, the position information of the pixels in the target area can be obtained and normalized.

[0226] In one possible implementation, the target object is the human body.

[0227] In one possible implementation, the positional information of pixels in the image to be processed includes the coordinates of the pixels in the image coordinate system, which is constructed based on the image to be processed.

[0228] For instructions on step 1201, please refer to [link / reference]. Figure 8 The relevant descriptions of step 801 in the illustrated embodiment will not be repeated here.

[0229] 1202. The pose parameters are obtained by processing the image to be processed and the position information of the pixels in the image to be processed through the model to be trained.

[0230] After obtaining the position information of pixels in the image to be processed and the normalized image to be processed, the position information of pixels in the image to be processed and the normalized image to be processed can be input into the model to be trained. The model to be trained will process the target region of the image to be processed and the position information of pixels in the normalized image to obtain the pose parameters.

[0231] In one possible implementation, the method further includes: processing the target region of the image to be processed and the position information of the pixels in the normalized image to be processed through a pose recognition model to obtain shape parameters and displacement parameters, and the pose parameters, shape parameters and displacement parameters are used together to obtain the pose of the target object.

[0232] For instructions on step 1202, please refer to [link / reference]. Figure 8 The relevant descriptions of step 802 in the illustrated embodiment will not be repeated here.

[0233] 1203. Based on the attitude parameters, obtain the predicted attitude of the target object.

[0234] After obtaining the attitude parameters, shape parameters, and displacement parameters, calculations can be performed based on these parameters to obtain the predicted attitude of the target object. It should be noted that the predicted attitude of the target object can be represented by multiple predicted 3D key points.

[0235] In one possible implementation, the predicted pose of the target object includes the predicted orientation of the target object in the camera coordinate system and the predicted limb behavior of the target object in the camera coordinate system, which is constructed based on the camera that captured the image to be processed.

[0236] For instructions on step 1203, please refer to [link / reference]. Figure 10 The relevant description of step 1003 in the illustrated embodiment will not be repeated here.

[0237] 1204. Based on the predicted pose and the true pose of the target object, the model to be trained is trained to obtain the pose recognition model.

[0238] After obtaining the predicted pose of the target object, since the true pose of the target object is known, the model to be trained can be trained based on the predicted pose and the true pose of the target object, resulting in the following: Figure 8 The pose recognition model in the illustrated embodiment.

[0239] In one possible implementation, the model to be trained is trained based on the predicted pose and the true pose of the target object to obtain a pose recognition model; the model to be trained is trained based on the predicted projection result of the predicted pose of the target object on the image to be processed and the true projection result of the true pose of the target object on the image to be processed to obtain a pose recognition model.

[0240] For instructions on step 1204, please refer to [link / reference]. Figure 10 The relevant description of step 1004 in the illustrated embodiment will not be repeated here.

[0241] The pose recognition model trained in this embodiment can perceive the position of a target object in the entire image and learn the relationship between that position and the pose of the target object, thereby capturing the pose of the target object more accurately. When the pose recognition model processes images, its input includes not only the target image but also the positional information of pixels in the target image. Therefore, the pose recognition model considers not only the influence of the image information of the target image itself on the pose of the target object but also the influence of the positional information of pixels in the target image on the pose of the target object. The factors considered are more comprehensive, so the pose of the target object obtained in this way has high accuracy, which helps to improve the accuracy of the motion information of the acquired target image.

[0242] The above is a detailed description of the model training method provided in the embodiments of this application. The following will introduce the pose recognition device and model training device provided in the embodiments of this application. Figure 13 A schematic diagram of the posture recognition device provided in the embodiments of this application is shown below. Figure 13 As shown, the device includes:

[0243] Acquisition module 1301 is used to acquire the target image;

[0244] The processing module 1302 is used to process the target region of the target image and the position information of the target region in the target image through the pose recognition model to obtain pose parameters. The target region is the region where the target object is located, and the pose parameters are used to obtain the pose of the target object.

[0245] In this embodiment, after acquiring the target image, the target region where the target object is located and its position information within the target image can be input into the pose recognition model. The pose recognition model processes the target region and its position information to obtain pose parameters. Therefore, the pose of the target object can be obtained based on these pose parameters. In the aforementioned process, the input to the pose recognition model includes not only the cropped target region but also its position information within the target image. Thus, when processing the image, the pose recognition model considers not only the influence of the target region's image information on the target object's pose but also the influence of the target region's position information within the target image on the target object's pose. This comprehensive consideration results in a more accurate target object pose, thereby improving the accuracy of the motion information obtained from the target image.

[0246] In one possible implementation, the location information includes the coordinates of the center point of the target region in the image coordinate system and the size of the target region, the image coordinate system being constructed based on the target image.

[0247] In one possible implementation, the location information includes the coordinates of the vertices of the target region in an image coordinate system, which is constructed based on the target image.

[0248] In one possible implementation, the pose recognition model is trained based on the predicted projection of the target object's predicted pose onto the image to be processed and the true projection of the target object's true pose onto the image to be processed.

[0249] In one possible implementation, the pose of the target object includes the orientation of the target object in the camera coordinate system and the limb behavior of the target object in the camera coordinate system, which is constructed based on the camera that captures the target image.

[0250] In one possible implementation, the processing module 1302 is further configured to process the target region of the target image and the position information of the target region in the target image through the pose recognition model to obtain shape parameters and displacement parameters. The pose parameters, shape parameters and displacement parameters are used together to obtain the pose of the target object.

[0251] In one possible implementation, the device further includes: a normalization module for normalizing the position information of the target region in the target image to obtain normalized position information; and a processing module 1302 for processing the target region of the target image and the normalized position information through a pose recognition model to obtain pose parameters.

[0252] In one possible implementation, the target object is the human body.

[0253] Figure 14 This is another structural schematic diagram of the posture recognition device provided in the embodiments of this application, as shown below. Figure 14 As shown, the device includes:

[0254] Acquisition module 1401 is used to acquire the target image;

[0255] The processing module 1402 is used to process the target image and the position information of the pixels in the target image through the pose recognition model to obtain pose parameters. The pose parameters are used to obtain the pose of the target object contained in the target image.

[0256] In this embodiment, after acquiring the target image, the target image and the position information of its pixels can be input into a pose recognition model. The pose recognition model processes the target image and the position information of its pixels to obtain pose parameters. Therefore, the pose of the target object presented in the target image can be obtained based on the pose parameters. In the aforementioned process, the input to the pose recognition model includes not only the target image but also the position information of its pixels. Therefore, when processing the image, the pose recognition model considers not only the influence of the image information of the target image itself on the pose of the target object but also the influence of the position information of the pixels in the target image on the pose of the target object. The factors considered are comprehensive, so the pose of the target object obtained in this way has high accuracy, which helps to improve the accuracy of the motion information of the acquired target image.

[0257] In one possible implementation, the location information includes the coordinates of the pixel in an image coordinate system constructed based on the target image.

[0258] In one possible implementation, the pose recognition model is trained based on the predicted projection of the target object's predicted pose onto the image to be processed and the true projection of the target object's true pose onto the image to be processed.

[0259] In one possible implementation, the pose of the target object includes the orientation of the target object in the camera coordinate system and the limb behavior of the target object in the camera coordinate system, which is constructed based on the camera that captures the target image.

[0260] In one possible implementation, the processing module 1402 is further configured to process the target image and the position information of the pixels in the target image through the pose recognition model to obtain shape parameters and displacement parameters. The pose parameters, shape parameters and displacement parameters are used together to obtain the pose of the target object.

[0261] In one possible implementation, the device further includes: a normalization module for normalizing the position information of pixels in the target image to obtain normalized position information; and a processing module 1402 for processing the target image and the normalized position information through a pose recognition model to obtain pose parameters.

[0262] In one possible implementation, the target object is the human body.

[0263] Figure 15 Another schematic diagram of the model training device provided in the embodiments of this application is shown below. Figure 15 As shown, the device includes:

[0264] The first acquisition module 1501 is used to acquire the image to be processed;

[0265] Processing module 1502 is used to process the target region of the image to be processed and the position information of the target region in the image to be processed through the model to be trained, so as to obtain the pose parameters. The target region is the region where the target object is located.

[0266] The second acquisition module 1503 is used to acquire the predicted pose of the target object based on the pose parameters;

[0267] Training module 1504 is used to train the model to be trained based on the predicted pose and the true pose of the target object, so as to obtain the pose recognition model.

[0268] The pose recognition model obtained in this embodiment allows the model to perceive the position of a target object in the entire image and learn the relationship between that position and the target object's pose, thereby capturing the target object's pose more accurately. When the pose recognition model is used for image processing, its input includes not only the cropped target region but also the positional information of the target region within the target image. Therefore, the pose recognition model considers not only the influence of the target region's own image information on the target object's pose but also the influence of the target region's positional information within the target image on the target object's pose. This comprehensive consideration results in higher accuracy of the target object's pose obtained through this method, thus improving the accuracy of the motion information obtained from the target image.

[0269] In one possible implementation, the training module 1504 is used to train the training model based on the predicted projection result of the target object's predicted pose on the image to be processed and the true projection result of the target object's true pose on the image to be processed, so as to obtain the pose recognition model.

[0270] In one possible implementation, the location information includes the coordinates of the center point of the target region in the image coordinate system and the size of the target region, the image coordinate system being constructed based on the image to be processed.

[0271] In one possible implementation, the location information includes the coordinates of the vertices of the target region in an image coordinate system, which is constructed based on the image to be processed.

[0272] In one possible implementation, the predicted pose of the target object includes the predicted orientation of the target object in the camera coordinate system and the predicted limb behavior of the target object in the camera coordinate system, which is constructed based on the camera that captured the image to be processed.

[0273] In one possible implementation, the processing module 1502 is further configured to process the target region of the image to be processed and the position information of the target region in the image to be processed through the pose recognition model to obtain shape parameters and displacement parameters. The pose parameters, shape parameters and displacement parameters are used together to obtain the pose of the target object.

[0274] In one possible implementation, the device further includes: a normalization module for normalizing the position information of the target region in the image to be processed to obtain normalized position information; and a processing module 1502 for processing the target region of the image to be processed and the normalized position information through the model to be trained to obtain pose parameters.

[0275] In one possible implementation, the target object is the human body.

[0276] Figure 16 Another schematic diagram of the model training device provided in the embodiments of this application is shown below. Figure 16 As shown, the device includes:

[0277] The first acquisition module 1601 is used to acquire the image to be processed;

[0278] The processing module 1602 is used to process the image to be processed and the position information of the pixels in the image to be processed through the model to be trained, so as to obtain the pose parameters.

[0279] The second acquisition module 1603 is used to acquire the predicted pose of the target object based on the pose parameters;

[0280] Training module 1604 is used to train the model to be trained based on the predicted pose and the true pose of the target object, so as to obtain the pose recognition model.

[0281] The pose recognition model obtained in this embodiment allows the model to perceive the position of a target object in the entire image and learn the relationship between that position and the pose of the target object, thereby capturing the pose of the target object more accurately. When the pose recognition model processes an image, its input includes not only the target image but also the positional information of the pixels within the target image. Therefore, the pose recognition model considers not only the influence of the image information itself on the pose of the target object but also the influence of the positional information of the pixels within the target image on the pose of the target object. This comprehensive consideration results in higher accuracy of the target object's pose obtained through this method, which in turn improves the accuracy of the motion information obtained from the target image.

[0282] In one possible implementation, the training module 1604 is used to train the training model based on the predicted projection result of the target object's predicted pose on the image to be processed and the true projection result of the target object's true pose on the image to be processed, so as to obtain the pose recognition model.

[0283] In one possible implementation, the location information includes the coordinates of the pixel in an image coordinate system constructed based on the image to be processed.

[0284] In one possible implementation, the predicted pose of the target object includes the predicted orientation of the target object in the camera coordinate system and the predicted limb behavior of the target object in the camera coordinate system, which is constructed based on the camera that captured the image to be processed.

[0285] In one possible implementation, the processing module 1602 is further configured to process the target region of the image to be processed and the position information of the pixels in the image to be processed through the pose recognition model to obtain shape parameters and displacement parameters. The pose parameters, shape parameters and displacement parameters are used together to obtain the pose of the target object.

[0286] In one possible implementation, the device further includes: a normalization module for normalizing the position information of pixels in the image to be processed to obtain normalized position information; and a processing module 1602 for processing the image to be processed and the normalized position information through the model to be trained to obtain pose parameters.

[0287] In one possible implementation, the target object is the human body.

[0288] It should be noted that the information interaction and execution process between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of this application, and the resulting technical effects are the same as those of the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in the embodiment of this application, and it will not be repeated here.

[0289] This application also relates to an execution device. Figure 17 This is a schematic diagram of the execution device provided in an embodiment of this application. Figure 17 As shown, the execution device 1700 can specifically manifest as a mobile phone, tablet, laptop, smart wearable device, server, etc., without limitation here. Among them, the execution device 1700 can be deployed with... Figure 13 or Figure 14 The image classification device described in the corresponding embodiment is used to implement Figure 6 or Figure 8This corresponds to the posture recognition function in the embodiment. Specifically, the execution device 1700 includes: a receiver 1701, a transmitter 1702, a processor 1703, and a memory 1704 (wherein the execution device 1700 may have one or more processors 1703). Figure 17 (Taking a processor as an example), processor 1703 may include application processor 17031 and communication processor 17032. In some embodiments of this application, receiver 1701, transmitter 1702, processor 1703 and memory 1704 may be connected via a bus or other means.

[0290] Memory 1704 may include read-only memory and random access memory, and provides instructions and data to processor 1703. A portion of memory 1704 may also include non-volatile random access memory (NVRAM). Memory 1704 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0291] Processor 1703 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0292] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 1703. Processor 1703 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 1703 or by instructions in software form. Processor 1703 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 1703 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 1704. Processor 1703 reads the information in memory 1704 and, in conjunction with its hardware, completes the steps of the above method.

[0293] Receiver 1701 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 1702 can be used to output digital or character information through the first interface; transmitter 1702 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1702 may also include a display device such as a display screen.

[0294] In one embodiment of this application, the processor 1703 is used to... Figure 6 or Figure 8 The pose recognition model in the corresponding embodiment performs pose recognition on the image.

[0295] This application also relates to a training device. Figure 18 This is a schematic diagram of the structure of a training device provided in an embodiment of this application. Figure 18As shown, the training device 1800 is implemented by one or more servers. The training device 1800 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1814 (e.g., one or more processors) and memory 1832, and one or more storage media 1830 (e.g., one or more mass storage devices) for storing application programs 1842 or data 1844. The memory 1832 and storage media 1830 can be temporary or persistent storage. The program stored in the storage media 1830 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the training device. Furthermore, the CPU 1814 may be configured to communicate with the storage media 1830 and execute the series of instruction operations in the storage media 1830 on the training device 1800.

[0296] The training device 1800 may also include one or more power supplies 1826, one or more wired or wireless network interfaces 1850, one or more input / output interfaces 1858; or, one or more operating systems 1841, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0297] Specifically, the training equipment can perform Figure 10 or Figure 12 The model training method in the corresponding embodiment.

[0298] This application also relates to a computer storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0299] This application also relates to a computer program product that stores instructions that, when executed by a computer, cause the computer to perform steps as performed by the aforementioned execution device, or to perform steps as performed by the aforementioned training device.

[0300] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0301] For details, please refer to Figure 19 , Figure 19 This is a schematic diagram of the chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1900. The NPU 1900 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1903, which is controlled by the controller 1904 to retrieve matrix data from the memory and perform multiplication operations.

[0302] In some implementations, the arithmetic circuit 1903 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1903 is a two-dimensional pulsating array. The arithmetic circuit 1903 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1903 is a general-purpose matrix processor.

[0303] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1902 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1901 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1908.

[0304] Unified memory 1906 is used to store input and output data. Weight data is directly transferred to weight memory 1902 via Direct Memory Access Controller (DMAC) 1905. Input data is also transferred to unified memory 1906 via DMAC.

[0305] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1909.

[0306] The Bus Interface Unit (BIU) 1913 is used by the instruction fetch memory 1909 to fetch instructions from external memory, and also by the memory access controller 1905 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0307] The DMAC is mainly used to move input data from external memory DDR to unified memory 1906, or to weight data to weight memory 1902, or to input data to input memory 1901.

[0308] The vector computation unit 1907 includes multiple processing units that further process the output of the computation circuit 1903 when needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as Batch Normalization, pixel-level summation, and upsampling of the predicted label plane.

[0309] In some implementations, the vector computation unit 1907 can store the processed output vector in the unified memory 1906. For example, the vector computation unit 1907 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1903, such as linearly interpolating the predicted label plane extracted from the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1907 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 1903, for example, for use in subsequent layers of the neural network.

[0310] The instruction fetch buffer 1909 connected to the controller 1904 is used to store the instructions used by the controller 1904;

[0311] Unified memory 1906, input memory 1901, weighted memory 1902, and instruction fetch memory 1909 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0312] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0313] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0314] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0315] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0316] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A pose recognition method, characterized in that, The method includes: Acquire the target image; The target region of the target image and the position information of the target region in the target image are processed by the pose recognition model to obtain pose parameters. The target region is the region where the target object is located. The pose parameters are used to obtain the pose of the target object. The pose parameters include the rotation angle of the target object relative to the camera and the included angle between the joints of the target object.

2. The method according to claim 1, characterized in that, The location information includes the coordinates of the center point of the target region in the image coordinate system and the size of the target region, wherein the image coordinate system is constructed based on the target image; or... The location information includes the coordinates of the vertices of the target region in the image coordinate system, which is constructed based on the target image.

3. The method according to claim 1 or 2, characterized in that, The pose recognition model is trained based on the predicted projection of the target object's predicted pose onto the image to be processed and the actual projection of the target object's true pose onto the image to be processed.

4. The method according to any one of claims 1 to 3, characterized in that, The pose of the target object includes the orientation of the target object in the camera coordinate system and the limb behavior of the target object in the camera coordinate system, which is constructed based on the camera that captured the target image.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The target region of the target image and the position information of the target region in the target image are processed by the pose recognition model to obtain shape parameters and displacement parameters. The pose parameters, shape parameters and displacement parameters are used together to obtain the pose of the target object.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: The position information of the target region in the target image is normalized to obtain normalized position information; The process of processing the target region of the target image and the position information of the target region in the target image through the pose recognition model to obtain pose parameters includes: The pose parameters are obtained by processing the target region of the target image and the normalized position information through a pose recognition model.

7. A pose recognition method, characterized in that, The method includes: Acquire the target image; The target image and the position information of the pixels in the target image are processed by a pose recognition model to obtain pose parameters. The pose parameters are used to obtain the pose of the target object contained in the target image. The pose parameters include the rotation angle of the target object relative to the camera and the included angle between the joints of the target object itself.

8. The method according to claim 7, characterized in that, The location information includes the coordinates of the pixel in the image coordinate system, which is constructed based on the target image.

9. The method according to claim 7 or 8, characterized in that, The pose recognition model is trained based on the predicted projection of the target object's predicted pose onto the image to be processed and the actual projection of the target object's true pose onto the image to be processed.

10. The method according to any one of claims 7 to 9, characterized in that, The pose of the target object includes the orientation of the target object in the camera coordinate system and the limb behavior of the target object in the camera coordinate system, which is constructed based on the camera that captured the target image.

11. The method according to any one of claims 7 to 10, characterized in that, The method further includes: The target image and the position information of the pixels in the target image are processed by a pose recognition model to obtain shape parameters and displacement parameters. The pose parameters, shape parameters and displacement parameters are used together to obtain the pose of the target object.

12. The method according to any one of claims 7 to 11, characterized in that, The method further includes: The position information of pixels in the target image is normalized to obtain normalized position information; The process of processing the target image and the position information of pixels in the target image using a pose recognition model to obtain pose parameters includes: The pose parameters are obtained by processing the target image and the normalized position information using a pose recognition model.

13. A model training method, characterized in that, The method includes: Obtain the image to be processed; The target region of the image to be processed and the position information of the target region in the image to be processed are processed by the model to be trained to obtain pose parameters. The target region is the region where the target object is located. The pose parameters include the rotation angle of the target object relative to the camera and the angle between the joints of the target object itself. Based on the attitude parameters, the predicted attitude of the target object is obtained; Based on the predicted pose of the target object and the actual pose of the target object, the model to be trained is trained to obtain a pose recognition model.

14. The method according to claim 13, characterized in that, The step of training the model to be trained based on the predicted pose of the target object and the true pose of the target object to obtain the pose recognition model includes: Based on the predicted projection result of the target object's predicted pose onto the image to be processed and the actual projection result of the target object's true pose onto the image to be processed, the training model is trained to obtain a pose recognition model.

15. The method according to claim 13 or 14, characterized in that, The location information includes the coordinates of the center point of the target region in the image coordinate system and the size of the target region, wherein the image coordinate system is constructed based on the image to be processed; or... The location information includes the coordinates of the vertices of the target region in the image coordinate system, which is constructed based on the image to be processed.

16. A model training method, characterized in that, The method includes: Obtain the image to be processed; The training model processes the image to be processed and the position information of the pixels in the image to be processed to obtain pose parameters; the pose parameters include the rotation angle of the target object relative to the camera and the included angle between the joints of the target object itself. Based on the attitude parameters, the predicted attitude of the target object is obtained; Based on the predicted pose of the target object and the actual pose of the target object, the model to be trained is trained to obtain a pose recognition model.

17. The method according to claim 16, characterized in that, The step of training the model to be trained based on the predicted pose of the target object and the true pose of the target object to obtain the pose recognition model includes: Based on the predicted projection result of the target object's predicted pose onto the image to be processed and the actual projection result of the target object's true pose onto the image to be processed, the training model is trained to obtain a pose recognition model.

18. The method according to claim 16 or 17, characterized in that, The location information includes the coordinates of the pixels in the image coordinate system, which is constructed based on the image to be processed.

19. A posture recognition device, characterized in that, The device includes: The acquisition module is used to acquire the target image; The processing module is used to process the target region of the target image and the position information of the target region in the target image through a pose recognition model to obtain pose parameters. The target region is the region where the target object is located. The pose parameters are used to obtain the pose of the target object. The pose parameters include the rotation angle of the target object relative to the camera and the included angle between the joints of the target object itself.

20. The apparatus according to claim 19, characterized in that, The location information includes the coordinates of the center point of the target region in the image coordinate system and the size of the target region, wherein the image coordinate system is constructed based on the target image; or... The location information includes the coordinates of the vertices of the target region in the image coordinate system, which is constructed based on the target image.

21. The method according to claim 19 or 20, characterized in that, The pose recognition model is trained based on the predicted projection of the target object's predicted pose onto the image to be processed and the actual projection of the target object's true pose onto the image to be processed.

22. A posture recognition device, characterized in that, The device includes: The acquisition module is used to acquire the target image; The processing module is used to process the target image and the position information of the pixels in the target image through a pose recognition model to obtain pose parameters. The pose parameters are used to obtain the pose of the target object contained in the target image. The pose parameters include the rotation angle of the target object relative to the camera and the included angle between the joints of the target object itself.

23. The apparatus according to claim 22, characterized in that, The location information includes the coordinates of the pixel in the image coordinate system, which is constructed based on the target image.

24. The apparatus according to claim 22 or 23, characterized in that, The pose recognition model is trained based on the predicted projection of the target object's predicted pose onto the image to be processed and the actual projection of the target object's true pose onto the image to be processed.

25. A model training device, characterized in that, The device includes: The first acquisition module is used to acquire the image to be processed; The processing module is used to process the target region of the image to be processed and the position information of the target region in the image to be processed through the model to be trained, so as to obtain pose parameters. The target region is the region where the target object is located. The pose parameters include the rotation angle of the target object relative to the camera and the included angle between the joints of the target object itself. The second acquisition module is used to acquire the predicted pose of the target object based on the pose parameters; The training module is used to train the model to be trained based on the predicted pose of the target object and the actual pose of the target object, so as to obtain a pose recognition model.

26. The apparatus according to claim 25, characterized in that, The training module is used to train the model to be trained based on the predicted projection result of the predicted pose of the target object on the image to be processed and the real projection result of the real pose of the target object on the image to be processed, so as to obtain a pose recognition model.

27. The apparatus according to claim 25 or 26, characterized in that, The location information includes the coordinates of the center point of the target region in the image coordinate system and the size of the target region, wherein the image coordinate system is constructed based on the image to be processed; or... The location information includes the coordinates of the vertices of the target region in the image coordinate system, which is constructed based on the image to be processed.

28. A model training device, characterized in that, The device includes: The first acquisition module is used to acquire the image to be processed; The processing module is used to process the image to be processed and the position information of the pixels in the image to be processed through the model to be trained to obtain pose parameters; the pose parameters include the rotation angle of the target object relative to the camera and the included angle between the joints of the target object itself. The second acquisition module is used to acquire the predicted pose of the target object based on the pose parameters; The training module is used to train the model to be trained based on the predicted pose of the target object and the actual pose of the target object, so as to obtain a pose recognition model.

29. The apparatus according to claim 28, characterized in that, The training module is used to train the model to be trained based on the predicted projection result of the predicted pose of the target object on the image to be processed and the real projection result of the real pose of the target object on the image to be processed, so as to obtain a pose recognition model.

30. The apparatus according to claim 28 or 29, characterized in that, The location information includes the coordinates of the pixels in the image coordinate system, which is constructed based on the image to be processed.

31. A posture recognition device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the gesture recognition device performs the method as described in any one of claims 1 to 18.

32. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 18.

33. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 18.

Images