Human body posture estimation method, device, electronic device and readable storage medium

Abstract

The present application discloses a method, device, electronic device and readable storage medium for estimating human body posture. The method includes inputting the image to be processed into the human body posture estimation component to obtain the initial posture estimation information of each human body contained in the image to be processed and the offset information of each key point of each human body; the human body posture estimation component includes a human body posture preliminary estimation model, a heat map-guided encoder and a graph structure-based refinement module. The human body posture preliminary estimation model performs initial posture estimation on the image to be processed, and the heat map-guided encoder generates key point embedding based on its input human body key point feature information and multiple heat maps; the graph structure-based refinement module generates the offset information of each key point of the human body according to the key point embedding and the human body structure. For each human body contained in the image to be processed, according to the corresponding initial posture estimation information and the offset information of each human body key point, the posture information at the next moment can be efficiently estimated.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, device, electronic device and readable storage medium for estimating human posture. Background Art

[0002] With the rapid development of computer vision technology, 2D human pose estimation, also known as two-dimensional human pose estimation, as the most basic but also very challenging computer vision task, has also been rapidly developed. Human pose estimation is a technology that determines the precise pixel position of key points. It is a basic tool for solving other advanced tasks, such as behavior recognition, pose tracking, pedestrian re-identification, and human-computer interaction.

[0003] In recent years, due to the powerful expression ability of CNN (Convolutional Neural Network), 2D posture estimation based on CNN has made great progress. These multi-person posture estimation methods based on CNN can be divided into Bottom-up method and Top-down method. The so-called Top-down method for human posture estimation means: first obtain all human instances in the image through target detection, and then perform human posture estimation on each human instance. The existing technology usually performs human posture estimation based on the CNN model of heat map. The model based on heat map converts the human posture estimation problem into a heat map estimation problem, generates a heat map for each key point, and the heat value in the heat map represents the probability of the key point at the pixel position. The top-down method focuses on improving the network structure to improve the effectiveness of heat map regression. For example, the resolution of the output feature map can be improved by adding several deconvolutional layers (also known as deconvolution layers). This is a relatively simple but high-performance method.

[0004] It is understandable that the constraint relationship between key points naturally forms a graph structure, and the rational use of the dependency relationship between key points is conducive to the positioning of key points. However, CNN cannot process data with non-Euclidean structures. In layman's terms, the number of adjacent vertices of each vertex in the topological graph may be different, so it is impossible to use convolution kernels of the same size to perform convolution operations. This leads to the CNN-based model's lack of ability to capture the dependency relationship between key points and difficulty in processing graph-structured data. Compared with models based on convolutional neural networks, graph convolutional networks can more effectively capture the dependency relationship between key points when dealing with human posture modeling problems. Models based on graph convolutional networks perform human posture estimation tasks, but graph convolutional networks use 1D, or one-dimensional, keypoint embeddings as input, and keypoint embeddings are one-dimensional feature representations of key points. When generating key point embedding, related technologies directly flatten 2D features to obtain 1D key point embedding, which will result in a large dimension of key point embedding. For example, the size of the 2D feature is 64×48, which becomes 3072 after flattening, making the key point embedding contain redundant information. If the key point embedding contains more redundant information, the number of model parameters and the computational cost will increase a lot, which will result in the final human posture estimation efficiency and accuracy failing to meet the requirements.

[0005] In view of this, how to efficiently and accurately realize human posture estimation is a technical problem that technical personnel in the relevant field need to solve. Summary of the invention

[0006] The present application provides a method, device, electronic device and readable storage medium for estimating human body posture, which can realize human body posture estimation efficiently and accurately.

[0007] To solve the above technical problems, the embodiments of the present invention provide the following technical solutions:

[0008] An embodiment of the present invention provides a method for estimating a human body posture, comprising:

[0009] Pre-building a human posture estimation component, the human posture estimation component including a human posture preliminary estimation model, a heat map guided encoder and a graph structure based refinement module;

[0010] Inputting the image to be processed into the human body posture estimation component to obtain the initial posture estimation information of the human body contained in the image to be processed and the offset information of each key point;

[0011] For a human body contained in the image to be processed, determining the posture information at the next moment according to the initial posture estimation information and the offset information of each key point;

[0012] Among them, the human body posture preliminary estimation model performs initial posture estimation on the image to be processed, and outputs human body key point feature information and multiple heat maps to the heat map-guided encoder; the heat map-guided encoder generates a key point embedding with a dimension equal to the number of channels of the human body key point feature information according to the human body key point feature information and each heat map; the graph structure-based refinement module generates offset information for each key point of the human body according to the key point embedding and the human body structure.

[0013] Optionally, generating a key point embedding having a dimension equal to the number of channels of the key point feature information of the human body according to the key point feature information of the human body and each heat map includes:

[0014] Converting the key point feature information of the human body to obtain corresponding spatial features;

[0015] For each heat map, the values ​​of the pixel positions whose heat values ​​are less than the preset threshold in each heat map are updated according to the preset value update rules to obtain the corresponding target heat map;

[0016] Using the spatial features and the target heat maps, the features of each key point of the human body are encoded;

[0017] The key point embedding is obtained by performing matrix multiplication along the channel dimension on each target heat map and the spatial feature.

[0018] Optionally, updating the pixel position value whose thermal value is less than a preset threshold in each thermal map according to a preset value updating rule includes:

[0019] The value setting relationship is called to update the value of the pixel position whose thermal value is less than the preset threshold in each original thermal map; the value setting relationship is:

[0020]

[0021] Wherein, threshold is the preset threshold, H ij ' is the heat value of the pixel position j in the target heat map corresponding to the i-th key point, H ij is the thermal value of the pixel position j in the original thermal map corresponding to the i-th key point.

[0022] Optionally, the utilizing the spatial features and the target heat maps to encode the features of key points of the human body includes:

[0023] The encoding relation is called to encode the features of each key point of the human body; the encoding relation is:

[0024]

[0025] Where embedding∈RK×C , embedding is the key point embedding of a key point, embeddings is the key point embedding of all key points, || means connection on the channel dimension, h j ' is the thermal value of the pixel position at point j in the target thermal map, f j is the feature vector of each pixel position on the spatial feature along the channel direction, K is the total number of key points, i represents the i-th key point, C is the dimension of the key point embedding, H*W represents the number of regions of the spatial feature, and R is a real number.

[0026] Optionally, the graph structure-based refinement module includes multiple graph convolutional layers and fully connected layers;

[0027] Each graph convolutional layer is used to perform message passing between key points and update the key point embedding;

[0028] The fully connected layer is used to predict the offset of each key point.

[0029] Optionally, the graph structure-based refinement module includes multiple edge convolutional layers.

[0030] Another aspect of the present invention provides a human body posture estimation device, including:

[0031] A component construction module, used to pre-construct a human posture estimation component, wherein the human posture estimation component includes a human posture preliminary estimation model, a heat map-guided encoder, and a graph-structure-based refinement module;

[0032] A posture estimation module is used to input the image to be processed into the human posture estimation component to obtain the initial posture estimation information of the human body contained in the image to be processed and the offset information of each key point; for the human body contained in the image to be processed, the posture information at the next moment is determined according to the initial posture estimation information and the offset information of each key point; wherein the human posture preliminary estimation model performs initial posture estimation on the image to be processed, and outputs human key point feature information and multiple heat maps to the heat map-guided encoder; the heat map-guided encoder generates a key point embedding with a dimension equal to the number of channels of the human key point feature information according to the human key point feature information and each heat map; the graph structure-based refinement module generates the offset information of each key point of the human body according to the key point embedding and the human body structure.

[0033] Optionally, the posture estimation module is further used for:

[0034] The human body key point feature information is converted to obtain corresponding spatial features; for each heat map, the values ​​of the pixel positions whose thermal values ​​are less than a preset threshold in each heat map are updated according to preset value update rules to obtain the corresponding target heat map; the key point features of the human body are encoded using the spatial features and each target heat map; the key point embedding is obtained by performing matrix multiplication of each target heat map and the spatial features along the channel dimension.

[0035] An embodiment of the present invention further provides an electronic device, comprising a processor, wherein the processor is configured to implement the steps of the human body posture estimation method as described in any of the preceding items when executing a computer program stored in a memory.

[0036] Finally, an embodiment of the present invention further provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the human body posture estimation method as described in any of the preceding items are implemented.

[0037] The advantage of the technical solution provided by the present application is that the dimension of the 1D key point embedding finally generated by the key point encoder guided by the heat map is the number of channels of the feature map, which is generally a very small number such as 32, 48 or hundreds, while the length obtained by directly flattening is usually thousands or even tens of thousands, which is much smaller than the length obtained by directly flattening the feature map in the prior art, thereby effectively reducing the number of parameters and the amount of calculation in the subsequent part of the human posture estimation component, thereby improving the overall human posture estimation efficiency. The refinement module based on the graph structure can explicitly fuse the features of adjacent key points through graph convolution, transmit messages between key points, and finally predict the offset of each key point for refining the initial posture estimation information, which is conducive to improving the prediction accuracy of human posture, thereby realizing human posture estimation efficiently and accurately.

[0038] In addition, the embodiments of the present invention also provide corresponding implementation devices, electronic devices and readable storage media for the human body posture estimation method, which further makes the method more practical, and the devices, electronic devices and readable storage media have corresponding advantages.

[0039] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A schematic diagram of a flow chart of a method for estimating a human body posture provided by an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of a framework of an exemplary application scenario provided for an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of key point embedding and decoding guided by a heat map provided by an embodiment of the present invention;

[0044] Figure 4 A structural diagram of a specific implementation of a human body posture estimation device provided by an embodiment of the present invention;

[0045] Figure 5 A structural diagram of a specific implementation of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] The terms "first", "second", "third", "fourth", etc. in the specification and claims of this application and the above drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0048] After introducing the technical solutions of the embodiments of the present invention, various non-limiting implementation methods of the present application are described in detail below.

[0049] See first Figure 1 , Figure 1 A flow chart of a method for estimating a human body posture provided by an embodiment of the present invention, the embodiment of the present invention may include the following contents:

[0050] S101: Pre-build a human posture estimation component.

[0051] In this step, the human body posture estimation component includes a human body posture preliminary estimation model, a heat map guided encoder and a graph structure based refinement module. This component can be used in any heat map network, and can improve the detection performance of the heat map network with minimal parameters and computational cost. Among them, the human body posture preliminary estimation model can be any existing human body posture estimation model based on heat map, such as a CNN model based on heat map. The human body posture preliminary estimation model is used as the backbone network of the human body posture estimation component to perform human body posture estimation on the input image in the first stage, that is, to decode the heat map to obtain a rough human body posture prediction result. The human body posture estimation model generates a heat map corresponding to each key point of each human body contained in the input image. The thermal value in the heat map represents the probability of the key point at the pixel position. After that, each heat map is decoded to obtain the coordinates of the human body key points, and the coordinates of the human body key points are used as the key point feature information of each human body in the input image. The second stage of the human posture estimation component includes a heat map-guided encoder and a graph-structure-based refinement module. The heat map-guided encoder generates key point embeddings based on the human key point feature map and heat map generated in the first stage. The heat map-guided encoder of this embodiment generates semantic information containing key point corresponding areas, rather than key point embeddings for the entire image area, which reduces redundant information and reduces computational and parameter costs. Afterwards, the key point embeddings and the human body structure, or the graph structure constructed according to the constraint relationship between the human body key points, are input into the graph-structure-based refinement module. The graph-structure-based refinement module performs message passing between key points by constructing a graph convolutional network to capture the dependency between key points and refine the decoding results of the heat map, thereby obtaining a refined prediction of the human body posture of the input image.

[0052] S102: Input the image to be processed into a human body posture estimation component to obtain initial posture estimation information of the human body contained in the image to be processed and offset information of each key point of each human body.

[0053] In this step, the initial posture prediction information is the human posture prediction result achieved by the first stage of the human posture estimation component, that is, the decoding result of the thermal map of the human posture preliminary estimation model. After the image to be processed is input into the human posture estimation component, the processing flow of the image to be processed is as follows: the human posture preliminary estimation model performs initial posture estimation on the image to be processed, and outputs human key point feature information and multiple thermal maps to the encoder guided by the thermal map; the encoder guided by the thermal map generates a key point embedding with a dimension of the number of channels of human key point feature information according to the human key point feature information and each thermal map; and the key point embedding is input into the refinement module based on the graph structure, and the refinement module based on the graph structure generates the offset information of each key point of the human body according to the key point embedding and the human body structure. The key points of the human body can be determined based on the human body structure, and the graph structure can be constructed according to the constraint relationship of the key points of the human body. The key points of the human body are node positions related to the human body posture, such as shoulder joints, elbow joints, knee joints, ankle joints, etc., which can be determined by technicians in this field according to actual needs, which does not affect the implementation of this application.

[0054] That is, in this step, the human posture estimation component generates K heat maps {H1, H2, H3..., H K}, where K is the number of key points, and each heat map H k Represents the confidence map of the kth key point, and then decodes the heat map to obtain the coordinates of the key point p of the human body k . Then, the features and heatmaps generated in the first stage are input into the keypoint encoder guided by the heatmap to generate keypoint embeddings. The keypoint embeddings and the graph structure constructed according to the constraint relationship of the keypoints of the human body are then input into the graph-based refinement module. In this module, the message passing between keypoints is achieved through a series of graph convolutions. Finally, a fully connected layer is used to predict the offset of each keypoint.

[0055] S103: For the human body in the image to be processed, the posture information at the next moment is determined according to the initial posture estimation information and the offset information of each key point.

[0056] It is understandable that the image to be processed may contain only one human body instance or may include multiple human body instances. If the image to be processed includes multiple human body instances, the above step S102 inputs the image to be processed into the human body posture estimation component, and the human body posture estimation component outputs the initial posture estimation information of each human body contained in the image to be processed and the offset information of all key points of each human body. Based on this, in this step, for each human body instance, the human body posture can be predicted based on the initial posture estimation information of the previous step and the offset information of each key point, that is, the posture information of the human body instance at the next moment is determined.

[0057] In the technical solution provided by the embodiment of the present invention, the dimension of the 1D key point embedding finally generated by the key point encoder guided by the heat map is the number of channels of the feature map, which is generally a very small number such as 32, 48 or hundreds, while the length obtained by directly flattening is usually thousands or even tens of thousands, which is much smaller than the length obtained by directly flattening the feature map in the prior art, thereby effectively reducing the number of parameters and the amount of calculation in the subsequent part of the human posture estimation component, thereby improving the overall human posture estimation efficiency. The refinement module based on the graph structure can explicitly fuse the features of adjacent key points through graph convolution, transmit messages between key points, and finally predict the offset of each key point for refining the initial posture estimation information, which is conducive to improving the prediction accuracy of human posture, thereby realizing human posture estimation efficiently and accurately.

[0058] It should be noted that there is no strict order of execution between the steps in this application. As long as they comply with the logical order, these steps can be executed simultaneously or in a certain preset order. Figure 1 This is just a schematic and does not mean that this is the only execution order.

[0059] In the above embodiment, there is no limitation on how to calculate the key point embedding. In this embodiment, an optional calculation method for key point embedding is provided, such as Figure 2 and Figure 3 As shown, it may include the following:

[0060] Transform the key point feature information of the human body to obtain the corresponding spatial features;

[0061] For each heat map, the values ​​of the pixel positions whose heat values ​​are less than the preset threshold in each heat map are updated according to the preset value update rules to obtain the corresponding target heat map;

[0062] Using spatial features and heat maps of each target, the key points of the human body are encoded;

[0063] The key point embedding is obtained by performing matrix multiplication of each target heat map and spatial feature along the channel dimension.

[0064] In this embodiment, in order to make the key point embedding contain more effective semantic information, we use the heat map generated in the first stage as a guide to gather features from the area corresponding to the key point and discard features from irrelevant areas, such as the background. In this module, in order to better train the model, the features of the first stage are not used directly, but the features are converted to generate spatial features, which can be converted through the following relationship:

[0065] F spatial =φ(F);

[0066] Among them, F is the human key point feature information generated in the first stage, φ is the two-dimensional convolution layer with a convolution kernel of 1*1, that is, Conv2D 1x1, which converts the human key point feature information once to generate the corresponding spatial feature F spatial In order to avoid useless semantic information in the key point embedding aggregation, each heat map can be further processed. For a clearer description, this embodiment uses a target heat map to represent the processed heat map, and the heat map before processing is called the original heat map. The value of the pixel position whose heat value is less than the threshold in the original heat map is set to 0, that is, the value setting relationship is called to update the value of the pixel position whose heat value is less than the preset threshold in each original heat map; the value setting relationship can be expressed as:

[0067]

[0068] In the formula, threshold is the preset threshold, H ij ′ is the thermal value of the pixel position j in the target thermal map corresponding to the i-th key point, H ij is the thermal value of the pixel position j in the original thermal map corresponding to the i-th key point.

[0069] Afterwards, the key point features can be encoded based on the processed heat map with the target heat map and spatial features, that is, the encoding relationship is called to encode the key point features of the human body; the encoding relationship can be expressed as:

[0070]

[0071] Where embedding∈R K×C , embedding is the key point embedding of a key point, embeddings is the key point embedding of all key points, || means connection on the channel dimension, h j ′ is the thermal value of the pixel position j in the target thermal map, f j is the feature vector of each pixel position on the spatial feature along the channel direction, K is the total number of key points, i represents the i-th key point, C is the dimension of the key point embedding, H*W represents the number of regions of the spatial feature, and R is a real number. Optionally, C can be set to 32 or 48. This is equivalent to weighted fusion of the H*W regions of the spatial feature, where the weight is the thermal value of the position corresponding to the pixel, and the thermal value of most regions is 0. Figure 3 As shown. Each key point corresponds to a heat map, which guides the generation of key point embedding, so that each key point focuses more on an area rather than the entire image range, reducing the fusion of redundant information. This embodiment can be implemented by performing matrix multiplication of the processed heat map and the spatial feature along the channel dimension, and the formula is as follows:

[0072]

[0073] Among them, H∈R K×(H×W) , which will flatten the heat map. (H×W)×C , that is, first arrange the dimensions of the spatial features into H×W×C, and then flatten the features.

[0074] It can be seen from the above that the key point encoder guided by the heat map of this embodiment is essentially equivalent to the matrix operation of the heat map and the feature map. The dimension of the 1D key point embedding generated in the end is the number of channels of the feature map, and the number of channels of the features generated by most heat map networks is much smaller than the length obtained by directly flattening the feature map. The number of channels is generally 32, 48 or hundreds of small numbers, while the length obtained by directly flattening is usually thousands or even tens of thousands, which will directly lead to an increase in the number of parameters and the amount of calculation in the subsequent part of the network. Furthermore, the key point embedding is guided by the heat map, and only the features around the key points are integrated, while the features of irrelevant areas are discarded. At the same time, it is simple to implement and can be achieved through matrix multiplication without the need for additional design.

[0075] In the above embodiment, the structure of the graph-based refinement module is not limited. In this embodiment, an optional structure of the graph-based refinement module is provided, such as Figure 2 As shown, it may include the following:

[0076] The graph-based refinement module includes multiple graph convolutional layers and fully connected layers; each graph convolutional layer is used to perform message passing between key points and update the key point embedding; the fully connected layer is used to predict the offset of each key point.

[0077] In this embodiment, the connection relationship between the key points of the human body can naturally form a graph. Based on this, this embodiment can use a graph convolutional network to explicitly model the relationship between the key points. The key point graph G = {V, E} represents a graph, where V represents a set of human key points, V = {v i} i＝1：K , for each point v i , all have their own characteristics x i , x i The keypoint embedding generated for the i-th keypoint. E represents a set of edges, E = {e i，j} i＝1：K，j＝1：k , represents the pairwise relationship between two key points. Figure 2As shown, given a graph structure, message passing between key points can be performed through a series of graph convolutions to update the key point embedding. As an optional implementation, the graph convolution network can adopt edge convolution, namely EdgeConv, that is, the refinement module based on the graph structure may include multiple edge convolution layers. Of course, technical personnel in the relevant field can also choose other graph convolutions to replace EdgeConv according to actual needs. In the EdgeConv layer, a fully connected layer is used to map the connected key points to obtain edge features, and the key point embedding is updated by aggregating the associated edge features. Finally, the offset of the key point is predicted by the fully connected layer of the refinement module based on the graph structure. The final prediction result pk of the kth key point is the sum of the heat map decoding result obtained in the first stage and the offset prediction in the second stage, that is, the posture estimation result of the kth key point of the human body contained in the image to be processed can be:

[0078] p k =D(H k )+o k

[0079] Where D is the decoding operation, o k is the offset information of the kth key point.

[0080] As can be seen from the above, this embodiment can explicitly fuse the features of adjacent key points through graph convolution and pass messages between key points. The graph-based refinement module predicts the offset of the key point, which is used to refine the decoding results of the heat map of the first stage of the framework, thereby improving the prediction accuracy of the model. Since the dimension of the key point embedding is set small, the parameter amount and computational cost of the heat map-guided encoder and the graph-based refinement module are very small, which can be ignored compared to the heat map network of the first stage.

[0081] The embodiment of the present invention also provides a corresponding device for the human body posture estimation method, which further makes the method more practical. Among them, the device can be described from the perspective of functional modules and hardware. The human body posture estimation device provided by the embodiment of the present invention is introduced below. The human body posture estimation device described below and the human body posture estimation method described above can be referred to each other.

[0082] From the perspective of functional modules, see Figure 4 , Figure 4 A structural diagram of a human body posture estimation device provided by an embodiment of the present invention in a specific implementation manner, the device may include:

[0083] The component construction module 401 is used to pre-construct a human posture estimation component, which includes a human posture preliminary estimation model, a heat map-guided encoder, and a graph-structure-based refinement module.

[0084] The posture estimation module 402 is used to input the image to be processed into the human posture estimation component to obtain the initial posture estimation information of each human body contained in the image to be processed and the offset information of each key point of each human body; for each human body contained in the image to be processed, according to the corresponding initial posture estimation information and the offset information of each key point, the posture information of the next moment is determined; wherein, the human posture preliminary estimation model performs initial posture estimation on the image to be processed, and outputs the human body key point feature information and multiple heat maps to the heat map guided encoder; the heat map guided encoder generates a key point embedding with a dimension equal to the number of channels of the human body key point feature information according to the human body key point feature information and each heat map; the graph structure based refinement module generates the offset information of each key point of the human body according to the key point embedding and the human body structure.

[0085] Optionally, in some implementations of the present embodiment, the posture estimation module 402 may also be used to: convert the feature information of key points of the human body to obtain corresponding spatial features; for each heat map, update the values ​​of the pixel positions whose thermal values ​​are less than a preset threshold in each heat map according to a preset value update rule to obtain a corresponding target heat map; encode the features of key points of the human body using spatial features and target heat maps; and obtain key point embedding by performing matrix multiplication of each target heat map and spatial features along the channel dimension.

[0086] As an optional implementation of the above embodiment, the above posture estimation module 402 can be further used to: call the value setting relationship to update the value of the pixel position whose thermal value in each original thermal map is less than the preset threshold; the value setting relationship is:

[0087]

[0088] In the formula, threshold is the preset threshold, H ij ' is the heat value of the pixel position j in the target heat map corresponding to the i-th key point, H ij is the thermal value of the pixel position j in the original thermal map corresponding to the i-th key point.

[0089] As another optional implementation of the above embodiment, the above posture estimation module 402 can be further used to: call the coding relationship to encode the key point features of the human body; the coding relationship is:

[0090]

[0091] Where embedding∈R K×C , embedding is the key point embedding of a key point, embeddings is the key point embedding of all key points, || means connection on the channel dimension, h j' is the thermal value of the pixel position at point j in the target thermal map, f j is the feature vector of each pixel position on the spatial feature along the channel direction, K is the total number of key points, i represents the i-th key point, C is the dimension of the key point embedding, H*W represents the number of regions of the spatial feature, and R is a real number.

[0092] Optionally, in some implementations of this embodiment, the above-mentioned graph structure-based refinement module may include multiple graph convolutional layers and fully connected layers;

[0093] Each graph convolutional layer is used to perform message passing between key points and update the key point embedding; the fully connected layer is used to predict the offset of each key point.

[0094] As an optional implementation of the above embodiment, the graph structure-based refinement module may include multiple edge convolution layers.

[0095] The functions of the functional modules of the human body posture estimation device described in the embodiment of the present invention can be specifically implemented according to the method in the above method embodiment. The specific implementation process can refer to the relevant description of the above method embodiment, which will not be repeated here.

[0096] It can be seen from the above that the embodiments of the present invention can achieve human body posture estimation efficiently and accurately.

[0097] The human body posture estimation device mentioned above is described from the perspective of functional modules. Furthermore, the present application also provides an electronic device, which is described from the perspective of hardware. Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application in one implementation manner. Figure 5 As shown, the electronic device includes a memory 50 for storing a computer program; and a processor 51 for implementing the steps of the human body posture estimation method mentioned in any of the above embodiments when executing the computer program.

[0098] Among them, the processor 51 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the processor 51 may also be a controller, a microcontroller, a microprocessor or other data processing chip. The processor 51 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 51 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 51 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 51 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0099] The memory 50 may include one or more computer-readable storage media, which may be non-transitory. The memory 50 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. The memory 50 may be an internal storage unit of an electronic device in some embodiments, such as a hard disk of a server. The memory 50 may also be an external storage device of an electronic device in other embodiments, such as a plug-in hard disk equipped on a server, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. Further, the memory 50 may also include both an internal storage unit of an electronic device and an external storage device. The memory 50 may not only be used to store application software and various types of data installed in the electronic device, such as: the code of the program for executing the vulnerability processing method, etc., but may also be used to temporarily store data that has been output or is to be output. In this embodiment, the memory 50 is at least used to store the following computer program 501, wherein, after the computer program is loaded and executed by the processor 51, the relevant steps of the human posture estimation method disclosed in any of the aforementioned embodiments can be implemented. In addition, the resources stored in the memory 50 may also include an operating system 502 and data 503, etc., and the storage method may be temporary storage or permanent storage. The operating system 502 may include Windows, Unix, Linux, etc. The data 503 may include but is not limited to data corresponding to the human body posture estimation result, etc.

[0100] In some embodiments, the electronic device may further include a display screen 52, an input / output interface 53, a communication interface 54 or a network interface, a power supply 55 and a communication bus 56. Among them, the display screen 52 and the input / output interface 53 such as a keyboard belong to the user interface, and the optional user interface may also include a standard wired interface, a wireless interface, etc. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface. The communication interface 54 may optionally include a wired interface and / or a wireless interface, such as a WI-FI interface, a Bluetooth interface, etc., which is usually used to establish a communication connection between the electronic device and other electronic devices. The communication bus 56 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0101] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than those shown in the figure, for example, it may also include a sensor 57 for realizing various functions.

[0102] The functions of the functional modules of the electronic device described in the embodiment of the present invention can be specifically implemented according to the method in the above method embodiment. The specific implementation process can refer to the relevant description of the above method embodiment, which will not be repeated here.

[0103] It can be seen from the above that the embodiments of the present invention can achieve human body posture estimation efficiently and accurately.

[0104] It is understandable that if the human posture estimation method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electrically erasable programmable ROM, register, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, removable disk, CD-ROM, disk or optical disk and other media that can store program code.

[0105] Based on this, an embodiment of the present invention further provides a readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the human body posture estimation method described in any of the above embodiments are performed.

[0106] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the hardware disclosed in the embodiments, including devices and electronic devices, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0107] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0108] The above is a detailed introduction to a human body posture estimation method, device, electronic device and readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for estimating a human body posture, characterized in that: include: Pre-building a human posture estimation component, the human posture estimation component including a human posture preliminary estimation model, a heat map guided encoder and a graph structure based refinement module; Inputting the image to be processed into the human body posture estimation component to obtain the initial posture estimation information of the human body contained in the image to be processed and the offset information of each key point; For a human body contained in the image to be processed, determining the posture information at the next moment according to the initial posture estimation information and the offset information of each key point; Among them, the human body posture preliminary estimation model performs initial posture estimation on the image to be processed, and outputs human body key point feature information and multiple heat maps to the encoder guided by the heat map; the heat map-guided encoder generates a key point embedding with a dimension equal to the number of channels of the human body key point feature information according to the human body key point feature information and each heat map; the graph structure-based refinement module generates offset information of each key point of the human body according to the key point embedding and the human body structure; wherein the graph structure-based refinement module includes multiple graph convolution layers and fully connected layers, each graph convolution layer is used to perform message passing between key points and update the key point embedding, and the fully connected layer is used to predict the offset of each key point.

2. The method for estimating human body posture according to claim 1, characterized in that: The step of generating a key point embedding having a dimension equal to the number of channels of the key point feature information of the human body according to the key point feature information of the human body and each heat map comprises: Converting the key point feature information of the human body to obtain corresponding spatial features; For each heat map, the values ​​of the pixel positions whose heat values ​​are less than the preset threshold in each heat map are updated according to the preset value update rules to obtain the corresponding target heat map; Using the spatial features and the target heat maps, the features of each key point of the human body are encoded; The key point embedding is obtained by performing matrix multiplication along the channel dimension on each target heat map and the spatial feature.

3. The method for estimating human body posture according to claim 2, characterized in that: The updating of the pixel position values ​​of the thermal values ​​in each thermal map that are less than the preset threshold according to the preset value updating rule includes: The value setting relationship is called to update the value of the pixel position whose thermal value is less than the preset threshold in each original thermal map; the value setting relationship is: Wherein, threshold is the preset threshold, H ij ' is the heat value of the pixel position j in the target heat map corresponding to the i-th key point, H ij is the thermal value of the pixel position j in the original thermal map corresponding to the i-th key point.

4. The method for estimating human body posture according to claim 2, characterized in that: The method of encoding the key point features of the human body by using the spatial features and the target heat maps includes: The encoding relation is called to encode the features of each key point of the human body; the encoding relation is: Where embedding∈R K×C , embedding is the keypoint embedding of a keypoint, embeddings is the keypoint embedding of all keypoints, | | Indicates connection on the channel dimension, h j ' is the thermal value of the pixel position at point j in the target thermal map, f j is the feature vector of each pixel position on the spatial feature along the channel direction, K is the total number of key points, i represents the i-th key point, C is the dimension of the key point embedding, H*W represents the number of regions of the spatial feature, and R is a real number.

5. The method for estimating human body posture according to claim 1, characterized in that: The graph-based refinement module includes a plurality of edge convolutional layers.

6. A human body posture estimation device, characterized in that: include: A component construction module, used to pre-construct a human posture estimation component, wherein the human posture estimation component includes a human posture preliminary estimation model, a heat map-guided encoder, and a graph-structure-based refinement module; A posture estimation module, used to input the image to be processed into the human body posture estimation component to obtain the initial posture estimation information of the human body contained in the image to be processed and the offset information of each key point; For the human body contained in the image to be processed, the posture information at the next moment is determined according to the initial posture estimation information and the offset information of each key point; wherein the human body posture preliminary estimation model performs initial posture estimation on the image to be processed, and outputs human body key point feature information and multiple heat maps to the encoder guided by the heat map; the heat map-guided encoder generates a key point embedding with a dimension equal to the number of channels of the human body key point feature information according to the human body key point feature information and each heat map; the graph structure-based refinement module generates the offset information of each key point of the human body according to the key point embedding and the human body structure; wherein the graph structure-based refinement module includes multiple graph convolution layers and fully connected layers, each graph convolution layer is used to perform message passing between key points and update the key point embedding, and the fully connected layer is used to predict the offset of each key point.

7. The human body posture estimation device according to claim 6, characterized in that: The posture estimation module is further used for: The human body key point feature information is converted to obtain corresponding spatial features; for each thermal map, the values ​​of the pixel positions whose thermal values ​​are less than a preset threshold in each thermal map are updated according to a preset value update rule to obtain a corresponding target thermal map; the key point features of the human body are encoded using the spatial features and each target thermal map; The key point embedding is obtained by performing matrix multiplication along the channel dimension on each target heat map and the spatial feature.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor is used to implement the steps of the human body posture estimation method as claimed in any one of claims 1 to 5 when executing a computer program stored in the memory.

9. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the human body posture estimation method according to any one of claims 1 to 5 are implemented.

Images