A method, device, equipment and storage medium for multi-person posture estimation

Abstract

The embodiments of the present application disclose a multi-person posture estimation method, device, equipment and storage medium. The relevant embodiments can be applied to various scenarios such as artificial intelligence or intelligent transportation to improve positioning accuracy and reduce structural error rate. The method of the embodiment of the present application includes: calculating the position loss of each joint point feature in the joint point feature set to obtain a predicted loss value, dividing the joint point feature set into categories of joint pairs to obtain a joint pair relationship feature set, sampling the joint pair relationship feature set based on a sampling strategy to obtain reference joint pair features, reference joint pair positive sample features and reference joint pair negative sample features, calculating consistency loss based on the reference joint pair features, reference joint pair positive sample features and reference joint pair negative sample features to obtain a paired consistency loss value, adjusting the parameters of the posture estimation model based on the paired consistency loss value and the predicted loss value to obtain a target posture estimation model.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a multi-person posture estimation method, apparatus, device and storage medium. Background Art

[0002] Human pose estimation is the process of recovering human joints from an image or a video.

[0003] With the development of deep learning, although the CNN-based human posture estimation method has achieved good performance, the existing deep learning-based human posture estimation method generally adopts a fully supervised method based on deep learning, which learns the most accurate feature representation that is beneficial to regression by designing complex network structures or feature learning modules.

[0004] However, complex network structures or module designs not only introduce a large amount of computation and complexity, but also because the human body is very flexible, there are self-occlusions and complex postures (which may not appear in the training data at all); or facing complex environments such as foreground occlusions, changes in perspective, occlusions between individuals, etc.; or there are large differences in appearance between images, all of which bring great challenges to the accurate positioning of human joints. Summary of the invention

[0005] The embodiments of the present application provide a multi-person posture estimation method, apparatus, device and storage medium, which are used to perform position regression on the learned human body spatial feature expression, and at the same time, perform consistency comparison constraints on the joint pair relationship features between adjacent joint pairs by constructing the joint pair relationship features, so that the relationship features between joint pairs of the same category are more consistent, and the relationship features between joint pairs of different categories are more dissimilar, so that the posture estimation model can better learn the expression of the human body structure relationship, thereby improving the joint positioning accuracy and reducing the structural error rate.

[0006] An embodiment of the present application provides a method for multi-person posture estimation, including:

[0007] Based on the posture estimation model, a joint point feature set corresponding to each target object in multiple target objects is extracted from each sample image;

[0008] Based on the data annotation, the position loss is calculated for each joint point feature in the joint point feature set to obtain the predicted loss value corresponding to each joint point feature;

[0009] Based on the association relationship of human joints, the joint point feature set is divided into categories of joint pairs to obtain the joint pair relationship feature set corresponding to each joint pair category;

[0010] Based on the sampling strategy, the joint pair relationship feature set is sampled to obtain reference joint pair features, reference joint pair positive sample features, and reference joint pair negative sample features;

[0011] Based on the reference joint pair features, the reference joint pair positive sample features, and the reference joint pair negative sample features, the consistency loss is calculated to obtain the pairwise consistency loss value corresponding to each joint pair category;

[0012] The parameters of the pose estimation model are adjusted based on the pairwise consistency loss value and the prediction loss value to obtain the target pose estimation model.

[0013] On the other hand, the present application provides a multi-person posture estimation device, comprising:

[0014] An acquisition unit, used for extracting a joint point feature set corresponding to each target object in a plurality of target objects from each sample image based on a posture estimation model;

[0015] A processing unit, configured to calculate the position loss of each joint point feature in the joint point feature set based on the data annotation, and obtain a predicted loss value corresponding to each joint point feature;

[0016] The processing unit is further used to classify the joint point feature set into joint pairs based on the human body joint association relationship, and obtain a joint pair relationship feature set corresponding to each joint pair category;

[0017] A determination unit is used to sample the joint pair relationship feature set based on a sampling strategy to obtain reference joint pair features, reference joint pair positive sample features, and reference joint pair negative sample features;

[0018] The processing unit is further used to calculate the consistency loss based on the reference joint pair features, the reference joint pair positive sample features and the reference joint pair negative sample features to obtain a pairwise consistency loss value corresponding to each joint pair category;

[0019] The processing unit is also used to adjust the parameters of the posture estimation model based on the pairwise consistency loss value and the prediction loss value to obtain the target posture estimation model.

[0020] In a possible design, in an implementation of another aspect of the embodiment of the present application,

[0021] The processing unit is further used to classify the joint point feature set into joint categories based on the joint category labels to obtain a category joint feature set corresponding to each joint category;

[0022] The determination unit is further used to sample the category joint feature set based on the sampling strategy to obtain reference joint point features, reference joint point positive sample features, and reference joint point negative sample features;

[0023] The processing unit is further used to calculate the consistency loss based on the reference joint point features, the reference joint point positive sample features and the reference joint point negative sample features to obtain a single joint consistency loss value corresponding to each joint category;

[0024] The processing unit can be specifically used to: adjust the parameters of the posture estimation model based on the single-joint consistency loss value, the pairwise consistency loss value and the prediction loss value to obtain the target posture estimation model.

[0025] In a possible design, in an implementation of another aspect of the embodiment of the present application,

[0026] The processing unit is further used to map each joint point feature in all joint point feature sets to the target space;

[0027] The processing unit is also used to perform dimensionality reduction processing on each joint point feature in the target space to obtain a target dimension joint feature set;

[0028] The processing unit can be specifically used to: divide the target dimension joint feature set into categories of joint pairs based on the human body joint association relationship, and obtain a joint pair relationship feature set corresponding to each joint pair category;

[0029] The processing unit can be specifically used to: divide the target dimension joint feature set into joint categories based on the joint category labels, and obtain a category joint feature set corresponding to each joint category.

[0030] In a possible design, in an implementation of another aspect of the embodiment of the present application, the processing unit may be specifically used for:

[0031] From the target dimension joint feature set, determine the pairwise adjacent joint point features that satisfy the human joint association relationship, where each pairwise adjacent joint point feature corresponds to a joint pair category;

[0032] The joint pair relationship features are calculated based on the features of the two adjacent joint points to obtain the joint pair relationship feature set corresponding to each joint pair category.

[0033] In a possible design, in an implementation of another aspect of the embodiment of the present application, the determining unit may be specifically used to:

[0034] Taking any joint pair relationship feature in the joint pair relationship feature set as a reference joint pair feature;

[0035] The joint pair relationship features other than the reference joint pair features in the joint pair relationship feature set are used as the reference joint pair positive sample features;

[0036] From the set of other joint pair relationship features, a reference joint pair negative sample feature corresponding to the reference joint pair feature is determined.

[0037] In a possible design, in an implementation of another aspect of the embodiment of the present application, the determining unit may be specifically used to:

[0038] Calculate the first feature similarity between the reference joint pair feature and the joint pair relationship features in other joint pair relationship feature sets;

[0039] According to the first feature similarity, the reference joint pair negative sample features are determined.

[0040] In a possible design, in an implementation of another aspect of the embodiment of the present application, the determining unit may be specifically used to:

[0041] From the category joint feature set, determine the reference joint point features whose predicted loss values ​​meet the screening conditions;

[0042] The joint point features other than the reference joint point features in the category joint feature set are used as the reference joint point positive sample features;

[0043] From the set of joint features of other categories, determine the negative sample features of the reference joint points corresponding to the reference joint point features.

[0044] In a possible design, in an implementation of another aspect of the embodiment of the present application, the determining unit may be specifically used to:

[0045] Calculate the second feature similarity between the reference joint point feature and the joint point features in the joint feature set of other categories;

[0046] According to the second feature similarity, the negative sample feature of the reference joint point is determined.

[0047] In a possible design, in an implementation of another aspect of the embodiment of the present application, the processing unit may be specifically used for:

[0048] Through the decoding structure of the posture estimation model, each joint point feature in the joint point feature set is decoded to obtain the corresponding predicted position of each joint point feature in the sample image;

[0049] The position loss is calculated for the predicted position based on the data annotation to obtain the predicted loss value corresponding to each joint point feature.

[0050] In a possible design, in an implementation of another aspect of the embodiment of the present application, the acquisition unit may be specifically used to:

[0051] Each sample image is detected by the target detector of the pose estimation model to obtain the score and position of each target object;

[0052] Based on the score and position corresponding to each target object, the feature extractor of the posture estimation model is used to extract joint features of each target object to obtain a set of joint point features.

[0053] In a possible design, in an implementation of another aspect of the embodiment of the present application,

[0054] The acquisition unit is further used to acquire the image to be processed according to the posture estimation request when receiving the posture estimation request;

[0055] The processing unit is also used to input the image to be processed into the target posture estimation model, and output the posture estimation result corresponding to each target object through the target posture estimation model to obtain the human body posture estimation data corresponding to the image to be processed.

[0056] On the other hand, the present application provides a computer device, including: a memory, a processor, and a bus system;

[0057] Wherein, the memory is used to store programs;

[0058] The processor is used to implement the above-mentioned methods when executing the program in the memory;

[0059] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0060] Another aspect of the present application provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer is enabled to execute the above-mentioned methods.

[0061] It can be seen from the above technical solutions that the embodiments of the present application have the following beneficial effects:

[0062] Based on the posture estimation model, a joint point feature set corresponding to each target object in multiple target objects is extracted from each sample image. Then, based on data annotation, a position loss calculation can be performed on each joint point feature in the joint point feature set to obtain a predicted loss value corresponding to each joint point feature. Based on the human joint association relationship, the joint point feature set is divided into categories of joint pairs to obtain a joint pair relationship feature set corresponding to each joint pair category. Based on the sampling strategy, the joint pair relationship feature set is sampled to obtain reference joint pair features, reference joint pair positive sample features and reference joint pair negative sample features. Then, based on the reference joint pair features, the reference joint pair positive sample features and the reference joint pair negative sample features, a consistency loss calculation is performed to obtain a pairwise consistency loss value corresponding to each joint pair category. Based on the pairwise consistency loss value and the predicted loss value, the parameters of the posture estimation model can be adjusted to obtain a target posture estimation model. Through the above method, the joint point feature set in the image can be extracted by utilizing the backbone network of the posture estimation model to learn the spatial feature expression of the human body, and while performing position regression on the learned spatial feature expression of the human body, an additional feature mapping branch is used to map the joint point features of different human bodies into a feature space, and the joint pair relationship features between adjacent joint pairs are constructed. Based on the joint pair relationship features, the relationship between the joint pairs of different target objects in the image can be subjected to consistent comparison constraints, so that the relationship features between the joint pairs of the same category are more consistent, and the relationship features between the joint pairs of different categories are more dissimilar, so that the posture estimation model can better learn the expression of the human body structure relationship, thereby improving the joint positioning accuracy and reducing the structural error rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 is a schematic diagram of the architecture of a posture estimation control system in an embodiment of the present application;

[0064] Figure 2 This is a flow chart of an embodiment of a method for estimating multi-person postures in an embodiment of the present application;

[0065] Figure 3 is another embodiment flow chart of the method for estimating multi-person postures in the embodiment of the present application;

[0066] Figure 4 is another embodiment flow chart of the method for estimating multi-person postures in the embodiment of the present application;

[0067] Figure 5 is another embodiment flow chart of the method for estimating multi-person postures in the embodiment of the present application;

[0068] Figure 6 is another embodiment flow chart of the method for estimating multi-person postures in the embodiment of the present application;

[0069] Figure 7 is another embodiment flow chart of the method for estimating multi-person postures in the embodiment of the present application;

[0070] Figure 8 is another embodiment flow chart of the method for estimating multi-person postures in the embodiment of the present application;

[0071] Fig. 9 is another embodiment flow chart of the method for estimating multi-person postures in the embodiment of the present application;

[0072] Fig.10 is another embodiment flow chart of the method for estimating multi-person postures in the embodiment of the present application;

[0073] Fig.11 is another embodiment flow chart of the method for estimating multi-person postures in the embodiment of the present application;

[0074] Fig.12 is another embodiment flow chart of the method for estimating multi-person postures in the embodiment of the present application;

[0075] Fig.13 It is a schematic diagram of a principle flow chart of a multi-person posture estimation method in an embodiment of the present application;

[0076] Fig.14 1 is a schematic diagram of the principle of learning the relationship features of joint pairs in a method for estimating multi-person postures in an embodiment of the present application;

[0077] Fig.15 is a schematic diagram of an embodiment of a multi-person posture estimation device in an embodiment of the present application;

[0078] Fig.16 It is a schematic diagram of an embodiment of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0079] The embodiments of the present application provide a multi-person posture estimation method, apparatus, device and storage medium, which are used to perform position regression on the learned human body spatial feature expression, and at the same time, perform consistency comparison constraints on the joint pair relationship features between adjacent joint pairs by constructing the joint pair relationship features, so that the relationship features between joint pairs of the same category are more consistent, and the relationship features between joint pairs of different categories are more dissimilar, so that the posture estimation model can better learn the expression of the human body structure relationship, thereby improving the joint positioning accuracy and reducing the structural error rate.

[0080] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0081] 1. Convolutional Neural Network (CNN)

[0082] Convolutional neural networks are the foundation of deep learning. Fully connected neural networks usually take numerical values ​​as input. Since it is necessary to extract features from the image and sample them when processing image-related information, CNN can be used to integrate feature extraction, downsampling and fully connected neural networks to form a new network.

[0083] 2. (C, H, W)

[0084] C, H, and W are used to indicate the number of channels, height, and width corresponding to the feature map, respectively.

[0085] 3. Multi-person Pose Estimation (MPPE)

[0086] Multi-person pose estimation can be based on a top-down approach to solve the problems of inaccurate positioning of joint points and incorrect recognition structure in multi-person situations.

[0087] 4. Semi-supervised learning (SSL)

[0088] The basic setting of semi-supervised learning is that given a set of labeled examples L = {(x1, y1), (x2, y2), ..., (x|L|, y|L|)} from an unknown distribution and an unlabeled set U = {x1', x2', ..., x|U|'}, it is expected that the learned function f:X→Y can accurately predict the label y for example x. Here xi, xj'∈X are both d-dimensional vectors, yi∈Y is the label of example xi, |L| and |U| are the sizes of L and U, that is, the number of examples they contain.

[0089] It can be understood that in the specific implementation of the present application, related data such as sample images, joint point features and images to be processed are involved. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0090] It is understandable that the multi-person posture estimation method disclosed in this application involves artificial intelligence (AI) technology, and artificial intelligence technology is further introduced below. Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0091] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0092] Secondly, natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0093] Secondly, Machine Learning (ML) is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0094] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0095] It can be understood that the multi-person posture estimation method disclosed in the present application also specifically involves an Intelligent Vehicle Infrastructure Cooperative System (IVICS), and the Intelligent Vehicle Infrastructure Cooperative System is further introduced below. The Intelligent Vehicle Infrastructure Cooperative System is referred to as the Vehicle Infrastructure Cooperative System, which is a development direction of the Intelligent Transportation System (ITS). The vehicle-road cooperative system adopts advanced wireless communications and new generation Internet technologies to implement dynamic real-time information interaction between vehicles and roads in all directions, and carries out active vehicle safety control and road cooperative management based on the collection and integration of dynamic traffic information in all time and space, fully realizing the effective coordination of people, vehicles and roads, ensuring traffic safety, and improving traffic efficiency, thereby forming a safe, efficient and environmentally friendly road traffic system.

[0096] It should be understood that the multi-person posture estimation method provided in this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc., for human posture estimation through images to complete scenes such as movie animation, virtual reality, autonomous driving, video detection or motion analysis. As an example, for example, human body posture estimation is performed on image A to complete the classification of human body movements. As another example, for example, human body posture estimation is performed through image B to complete the detection of abnormal human behavior. As another example, for example, human body posture estimation is performed through image C to complete the 3D animation construction of image C.

[0097] In order to solve the above problems, this application proposes a multi-person posture estimation method, which is applied to Figure 1 The attitude estimation control system shown in Figure 1 , Figure 1 Schematic diagram of an architecture of a posture estimation control system in an embodiment of the present application. Figure 1 As shown, the server extracts a joint point feature set corresponding to each target object in multiple target objects from each sample image provided by the terminal device based on the posture estimation model, and then, based on the data annotation, the position loss calculation can be performed on each joint point feature in the joint point feature set to obtain the predicted loss value corresponding to each joint point feature, and based on the human body joint association relationship, the joint point feature set is divided into categories of joint pairs to obtain the joint pair relationship feature set corresponding to each joint pair category, and based on the sampling strategy, the joint pair relationship feature set is sampled to obtain reference joint pair features, reference joint pair positive sample features and reference joint pair negative sample features, and then, based on the reference joint pair features, the reference joint pair positive sample features and the reference joint pair negative sample features, the consistency loss calculation is performed to obtain the pairwise consistency loss value corresponding to each joint pair category, and the posture estimation model parameters can be adjusted based on the pairwise consistency loss value and the predicted loss value to obtain the target posture estimation model. Through the above method, the joint point feature set in the image can be extracted by utilizing the backbone network of the posture estimation model to learn the spatial feature expression of the human body, and while performing position regression on the learned spatial feature expression of the human body, an additional feature mapping branch is used to map the joint point features of different human bodies into a feature space, and the joint pair relationship features between adjacent joint pairs are constructed. Based on the joint pair relationship features, the relationship between the joint pairs of different target objects in the image can be subjected to consistent comparison constraints, so that the relationship features between the joint pairs of the same category are more consistent, and the relationship features between the joint pairs of different categories are more dissimilar, so that the posture estimation model can better learn the expression of the human body structure relationship, thereby improving the joint positioning accuracy and reducing the structural error rate.

[0098] Understandably, Figure 1 Only one type of terminal device is shown in the figure. In actual scenarios, more types of terminal devices may participate in the data processing process. Terminal devices include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, car terminals, etc. The specific number and type depend on the actual scenario and are not limited here. Figure 1 One server is shown in the figure, but in actual scenarios, multiple servers may also be involved, especially in the scenario of multi-model training interaction. The number of servers depends on the actual scenario and is not limited here.

[0099] It should be noted that in this embodiment, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal device and the server can be directly or indirectly connected through wired or wireless communication, and the terminal device and the server can be connected to form a blockchain network, which is not limited in this application.

[0100] Combined with the above introduction, the following will introduce the multi-person posture estimation method in this application, please refer to Figure 2 In one embodiment of the present application, a method for estimating multi-person postures includes:

[0101] In step S101, based on the posture estimation model, a joint point feature set corresponding to each target object in a plurality of target objects is extracted from each sample image;

[0102] In this embodiment, after obtaining several batches of sample images, object recognition and feature extraction can be performed on each sample image in each batch based on the framework of a conventional posture estimation model, so as to extract a set of joint point features corresponding to each target object in multiple target objects from each sample image.

[0103] The network architecture of the posture estimation model uses ResNet and HRNet as the backbone network, and other backbones include Res2Net, VGG or DenseNet, etc., which are not specifically limited here. The sample image can be specifically expressed as an image containing at least one full body of a human body, or an image of a part of a human body, etc., which are not specifically limited here. The target object can be used for a target detection frame containing a human body.

[0104] Specifically, Fig.13As shown, ResNet and HRNet can be used as the posture estimation model of the backbone network, and a two-stage processing can be performed on each sample image in each batch based on a top-down method. For example, in the first stage, the network structure that can be used to detect the target object in the posture estimation model, such as a human detector, can be used to detect each sample image in a batch to obtain a set of different human bodies, that is, multiple target objects. For each target object, each person can be regarded as a picture. Then, for multiple images in a batch of input, feature extraction can be performed through a feature extractor to extract multiple joint point features corresponding to each target object, and integrate them to obtain a joint point feature set.

[0105] In step S102, based on the data annotation, position loss calculation is performed on each joint point feature in the joint point feature set to obtain a predicted loss value corresponding to each joint point feature;

[0106] In this embodiment, after obtaining the joint point feature set, the data annotation information carried by each sample image can be used to predict the position of each joint point feature in the joint point feature set, and perform position loss calculation to obtain the predicted loss value corresponding to each joint point feature.

[0107] Among them, data labeling (groundtruth) means that in supervised learning, the data is labeled, usually in the form of (x, t), where x is used to represent the input data and t is the label. It can be understood that the correct t can usually be labeled as ground truth, and the wrong one can be marked as not.

[0108] Specifically, Fig.13 As shown, after obtaining the joint feature set, the decoding structure ψ of posture estimation can be used r , get the predicted position corresponding to each joint point feature, then use groundtruth to constrain the output predicted position, specifically by using classification constraints, regression constraints, or other constraint methods, which are not specifically limited here, that is, perform position loss calculation, specifically by using classification loss calculation, position loss calculation, or other loss calculation methods, which are not specifically limited here, to obtain the predicted loss value corresponding to each joint point feature

[0109] In step S103, based on the human body joint association relationship, the joint point feature set is divided into categories of joint pairs to obtain a joint pair relationship feature set corresponding to each joint pair category;

[0110] In this embodiment, if Fig.13As shown in the figure, due to the errors easily caused by the complex human body structure in the image, such as severe occlusion, in order to better learn the human body structure relationship and help correct human body structure errors such as occlusion, after obtaining the joint point feature set, a two-stage processing is performed on each sample image in each batch based on a top-down method. In the second stage, a two-branch processing is performed. One branch can classify each joint point feature in the joint point feature set into joint pairs according to the real human joint association relationship, so as to obtain the joint pair relationship feature set corresponding to each joint pair category, so that the joint pair relationship comparison consistency constraint can be performed on the obtained joint pair relationship feature set in the future, so as to better learn more robust joint features.

[0111] Among them, the human joint association relationship refers to the connection mode of the human joint (keypoint key point) itself. The human joint association relationship can be specifically expressed as the connection relationship between the elbow joint and the wrist joint of the arm, or the connection relationship between the ankle joint and the knee joint of the leg, etc., and can also be expressed as other connection relationships, which are not specifically limited here. On the contrary, it can be understood that there is no connection relationship between non-human joint association relationships such as the elbow joint of the arm and the ankle.

[0112] Specifically, Fig.14 As shown, based on the human joint association relationship, the joint point feature set is divided into category joint pairs. Specifically, the joint point features related to the category are obtained, and according to the distribution between adjacent joints, the pairwise adjacent joint point features that satisfy the human joint association relationship are obtained, and the relationship features between the two joint points are calculated based on the pairwise adjacent joint point features, that is, the joint pair relationship features. To obtain the joint pair relationship feature set corresponding to each joint pair category.

[0113] In step S104, based on the sampling strategy, the joint pair relationship feature set is sampled to obtain reference joint pair features, reference joint pair positive sample features, and reference joint pair negative sample features;

[0114] In this embodiment, after obtaining the joint pair relationship feature set, the joint pair relationship feature set can be sampled based on a preset difficult sampling strategy to obtain reference joint pair features, reference joint pair positive sample features and reference joint pair negative sample features, to achieve data enhancement and further enhance the joint expression, so that the relationship between joint features of different human body categories between images can be explored, so that more robust joint features can be better learned.

[0115] The reference joint pair feature refers to the more challenging joint pair relationship feature in the joint pair relationship feature set. The reference joint pair positive sample feature refers to the joint pair relationship feature that belongs to the same joint pair category as the reference joint pair feature and comes from other target objects. The reference joint pair negative sample feature refers to the joint pair relationship feature in the joint pair relationship feature set that belongs to a different joint pair category than the reference joint pair feature.

[0116] Specifically, after obtaining the joint pair relationship feature set, the joint pair relationship feature set can be sampled based on a preset difficult sampling strategy. Specifically, any joint pair relationship feature in the joint pair relationship feature set can be used as a reference joint pair feature, such as the joint pair feature composed of the elbow joint and wrist joint of the arm. Then, the joint pair relationship features in the joint pair relationship feature set except the reference joint pair features can be used as reference joint pair positive sample features, and the reference joint pair negative sample features corresponding to the reference joint pair features can be screened out from other joint pair relationship feature sets.

[0117] In step S105, consistency loss is calculated based on the reference joint pair features, the reference joint pair positive sample features, and the reference joint pair negative sample features to obtain a pairwise consistency loss value corresponding to each joint pair category;

[0118] In this embodiment, after obtaining the reference joint pair features, the reference joint pair positive sample features and the reference joint pair negative sample features, the consistency loss calculation can be performed based on the reference joint pair features, the reference joint pair positive sample features and the reference joint pair negative sample features to obtain the pairwise consistency loss values ​​corresponding to each joint pair category, which can realize the contrast constraints between the paired joint relationship representations, thereby learning better human body structure relationship representations.

[0119] Specifically, Fig.13 As shown in FIG. 1 , after obtaining the reference joint pair features, the reference joint pair positive sample features, and the reference joint pair negative sample features, the following formula (1) can be used to calculate the consistency loss based on the reference joint pair features, the reference joint pair positive sample features, and the reference joint pair negative sample features to obtain the pairwise consistency loss value corresponding to each joint pair category:

[0120]

[0121] Among them, L pair Used to represent the pairwise consistency loss value; P v It is used to represent the joint pair relationship feature set corresponding to a joint pair category; v is used to represent the reference joint pair feature; v + Used to represent the reference joint alignment sample features; v -Used to represent the reference joint pair negative sample features; N v It is used to represent the joint pair relationship feature set corresponding to other joint pair categories; τ is used to represent the temperature coefficient, which is a hyperparameter set according to actual application requirements. There is no specific restriction here, and it can usually be taken as 0.1.

[0122] In step S106, parameters of the pose estimation model are adjusted based on the pairwise consistency loss value and the prediction loss value to obtain a target pose estimation model.

[0123] In this embodiment, after obtaining the pairwise consistency loss value and the predicted loss value, the posture estimation model can be continuously iteratively optimized based on the pairwise consistency loss value and the predicted loss value until the model converges to obtain the target posture estimation model.

[0124] Specifically, Fig.13 As shown in the figure, after obtaining the pairwise consistency loss value and the prediction loss value, by combining the regression constraint on the predicted position, namely the prediction loss value, and the contrast consistency constraint on the representation of the structural relationship of the joint pair, namely the pairwise consistency loss value, based on the back propagation algorithm, the parameters of the posture estimation model are continuously iteratively optimized until the model converges to obtain the target posture estimation model. The expression of the joints can be enhanced by utilizing the consistency representation between the joint features related to the categories between images, so that no additional computational overhead is introduced during testing, thereby improving the detection performance of the model in an effective and efficient way, thereby achieving more accurate position positioning based on the target posture estimation model.

[0125] In an embodiment of the present application, a multi-person posture estimation method is provided. Through the above-mentioned method, a set of joint point features in an image can be extracted by utilizing the backbone network of a posture estimation model to learn the spatial feature expression of a human body, and while performing position regression on the learned spatial feature expression of a human body, an additional feature mapping branch is used to map the joint point features of different human bodies into a feature space, and joint pair relationship features between adjacent joint pairs are constructed. Based on the joint pair relationship features, consistency comparison constraints can be performed on the relationships between joint pairs of different target objects in the image, so that the relationship features between joint pairs of the same category are more consistent, and the relationship features between joint pairs of different categories are more dissimilar, so that the posture estimation model can better learn the expression of the structural relationship of the human body, thereby improving the accuracy of joint positioning and reducing the structural error rate.

[0126] Optionally, in the above Figure 2 Based on the corresponding embodiment, in another optional embodiment of the multi-person posture estimation method provided in the embodiment of the present application, as Figure 3As shown, step S106 adjusts the parameters of the posture estimation model based on the pairwise consistency loss value and the prediction loss value to obtain the target posture estimation model. The method further includes: steps S301 to S303; step S106 includes: step S304;

[0127] In step S301, based on the joint category labels, the joint point feature set is divided into joint categories to obtain a category joint feature set corresponding to each joint category;

[0128] In this embodiment, if Fig.13 As shown in the figure, in the process of feature extraction of sample images, it is easy to cause positioning errors (such as jitter) or missed detection due to insufficient features, resulting in a decrease in the accuracy of joint positioning. Therefore, in order to better learn the relationship between joint point features related to different human body categories, after obtaining the joint point feature set, a two-stage processing is performed on each sample image in each batch based on a top-down method. In the second stage, a two-branch processing is performed, and the other branch can divide the joint point feature set into joint categories based on the joint category label to obtain the category joint feature set corresponding to each joint category, so that the obtained category joint feature set can be subsequently subjected to single joint comparison consistency constraints to better learn robust joint features that are more dispersed between classes and more consistent within classes.

[0129] Specifically, Fig.14 As shown in FIG, based on the joint category label such as wrist joint, elbow joint or knee joint and other specific joint categories, it can also be represented as other joint categories, which are not specifically limited here. By combining the score and position corresponding to each joint point feature, the joint point feature set is divided into joint categories, and the feature representation f of the joint related to each category can be obtained. ki .

[0130] In step S302, based on the sampling strategy, the category joint feature set is sampled to obtain reference joint point features, reference joint point positive sample features, and reference joint point negative sample features;

[0131] In this embodiment, after obtaining the category joint feature set, the joint feature set can be sampled based on a preset difficult sampling strategy to obtain reference joint point features, reference joint point positive sample features, and reference joint point negative sample features, to achieve data enhancement, and further enhance the joint expression, so that the relationship between joint features of different human body categories between images can be explored, so that more robust joint features can be better learned.

[0132] The reference joint point feature refers to the more challenging joint point feature in the joint feature set. The reference joint point positive sample feature refers to the joint point feature that belongs to the same joint category as the reference joint point feature and comes from other target objects. The reference joint point negative sample feature refers to the joint point feature in the joint feature set that belongs to a different joint category than the reference joint point feature.

[0133] Specifically, after obtaining the category joint feature set, the category joint feature set can be sampled based on a preset difficulty sampling strategy. Specifically, the reference joint point features whose predicted loss values ​​meet the filtering conditions can be screened out from the category joint feature set, and the joint point features other than the reference joint point features in the category joint feature set can be used as the reference joint point positive sample features. Then, the reference joint point negative sample features corresponding to the reference joint point features can be screened out from other category joint feature sets.

[0134] In step S303, consistency loss calculation is performed based on the reference joint point features, the reference joint point positive sample features, and the reference joint point negative sample features to obtain a single joint consistency loss value corresponding to each joint category;

[0135] In this embodiment, after obtaining the reference joint point features, the reference joint point positive sample features and the reference joint point negative sample features, the consistency loss calculation can be performed based on the reference joint point features, the reference joint point positive sample features and the reference joint point negative sample features to obtain the single joint consistency loss value corresponding to each joint category, which can realize the contrast constraints between single joint representations, thereby learning robust joint features that are more dispersed between classes and more consistent within classes.

[0136] Specifically, Fig.13 As shown in FIG, after obtaining the reference joint point features, the reference joint point positive sample features and the reference joint point negative sample features, the following formula (2) can be used to calculate the consistency loss based on the reference joint point features, the reference joint point positive sample features and the reference joint point negative sample features to obtain the single joint consistency loss value corresponding to each joint category:

[0137]

[0138] Among them, L sin Used to represent the single joint consistency loss value; P i It is used to represent the category joint feature set corresponding to a joint category; i is used to represent the reference joint point feature; i + Used to represent the positive sample features of the reference joint point; i - Used to represent the negative sample features of the reference joint points; N iIt is used to represent the category joint feature set corresponding to other joint categories; τ is used to represent the temperature coefficient, which is a hyperparameter set according to actual application requirements. There is no specific restriction here, and it can usually be taken as 0.1.

[0139] In step S304, parameters of the posture estimation model are adjusted based on the single-joint consistency loss value, the pairwise consistency loss value, and the prediction loss value to obtain a target posture estimation model.

[0140] In this embodiment, after obtaining the single-joint consistency loss value, the pairwise consistency loss value and the predicted loss value, the posture estimation model can be continuously iteratively optimized based on the single-joint consistency loss value, the pairwise consistency loss value and the predicted loss value until the model converges to obtain the target posture estimation model. By enhancing the feature expressions related to each category, especially the learning of structural relationships, on the one hand, it can better solve the position jitter and missed detection errors that occur during positioning, and on the other hand, it can also better solve structural errors such as occlusion, thereby improving the accuracy of joint positioning and reducing the structural error rate to a certain extent.

[0141] Specifically, Fig.13 As shown in the figure, after obtaining the single-joint consistency loss value, the pairwise consistency loss value and the prediction loss value, by combining the regression constraint on the predicted position, i.e. the prediction loss value, and the single-joint consistency loss value and the pairwise consistency loss value, that is, based on the synergistic effect of the two comparative consistency constraints for the representation of the structural relationship of single joints and joint pairs, and then based on the back propagation algorithm, the parameters of the posture estimation model are continuously iteratively optimized until the model converges to obtain the target posture estimation model, so that more accurate position positioning can be achieved based on the target posture estimation model.

[0142] It is understandable that adjusting the parameters of the pose estimation model based on the single-joint consistency loss value, pairwise consistency loss value and prediction loss value not only achieves a good performance improvement at a relatively low cost under full supervision conditions, but also can still utilize the supervision information between images to a certain extent under semi-supervised conditions to make up for the performance loss caused by insufficient data annotation, thereby helping the pose estimation model to learn better joint features and structural relationships more novelly and efficiently.

[0143] Optionally, in the above Figure 3 Based on the corresponding embodiment, in another optional embodiment of the multi-person posture estimation method provided in the embodiment of the present application, as Figure 4As shown, before step S103 classifies the joint point feature set into joint pairs based on the human joint association relationship and obtains the joint pair relationship feature set corresponding to each joint pair category, the method further includes: step S401 to step S402; step S103 includes: step S403; step S301 includes: step 404;

[0144] In step S401, each joint point feature in all joint point feature sets is mapped to the target space;

[0145] In step S402, dimensionality reduction processing is performed on each joint point feature in the target space to obtain a target dimension joint feature set;

[0146] In step S403, based on the human body joint association relationship, the target dimension joint feature set is divided into categories of joint pairs to obtain a joint pair relationship feature set corresponding to each joint pair category;

[0147] In step S404, the target dimension joint feature set is divided into joint categories based on the joint category labels to obtain a category joint feature set corresponding to each joint category.

[0148] In this embodiment, based on the human body joint association relationship, the joint point feature set is divided into categories of joint pairs, and before obtaining the joint pair relationship feature set corresponding to each joint pair category, as shown in FIG. Fig.13 As shown, each joint point feature in all joint point feature sets can be mapped to the target space, and each joint point feature in the target space can be subjected to dimensionality reduction processing to obtain the target dimension joint feature set. The joint point features of fixed dimension in the same space can be obtained through spatial mapping and dimensionality reduction processing, so that the target dimension joint feature set can be subsequently divided into category joint pairs based on the human joint association relationship to better identify the joint pair relationship feature set corresponding to each joint pair category. Similarly, the target dimension joint feature set can be divided into joint categories based on the joint category labels to better obtain the category joint feature set corresponding to each joint category.

[0149] Specifically, Fig.13 As shown, the branch f using a feature map can be projMap each joint point feature in all joint point feature sets to the target space, transform the feature dimensions, that is, perform dimensionality reduction processing on each joint point feature in the target space, and output a fixed 256-dimensional feature representation to obtain the target dimension joint feature set. The target dimension joint feature set can be divided into category joint pairs based on the association relationship between human joints to better identify the joint pair relationship feature set corresponding to each joint pair category. Similarly, the target dimension joint feature set can be divided into joint categories based on joint category labels to better obtain the category joint feature set corresponding to each joint category.

[0150] Optionally, in the above Figure 4 Based on the corresponding embodiment, in another optional embodiment of the multi-person posture estimation method provided in the embodiment of the present application, as Figure 5 As shown, step S403 divides the target dimension joint feature set into joint pairs based on the human joint association relationship, and obtains a joint pair relationship feature set corresponding to each joint pair category, including:

[0151] In step S501, from the target dimension joint feature set, two-by-two adjacent joint point features that satisfy the human body joint association relationship are determined, wherein each two-by-two adjacent joint point feature corresponds to a joint pair category;

[0152] In step S502, joint pair relationship features are calculated based on the features of two adjacent joint points to obtain a set of joint pair relationship features corresponding to each joint pair category.

[0153] In this embodiment, after obtaining the target dimension joint feature set, the pairwise adjacent joint point features that satisfy the human joint association relationship can be screened out from the target dimension joint feature set, and the joint pair relationship features can be calculated based on the pairwise adjacent joint point features to obtain the joint pair relationship feature set corresponding to each joint pair category. Based on the real human body association relationship, the joint pair relationship features that conform to the real human body situation can be constructed, so that the posture estimation model can better learn more robust joint features.

[0154] Among them, each pairwise adjacent joint point feature corresponds to a joint pair category.

[0155] Specifically, Fig.14As shown in FIG. 1 , after obtaining the target dimension joint feature set, the pairwise adjacent joint feature that satisfies the human joint association relationship can be screened out based on the obtained category-related joint feature and the distribution between adjacent joints, such as the joint feature corresponding to the elbow and wrist joints of the arm. Then, the pairwise adjacent joint feature can be summed and averaged based on the following formula (3) to obtain the relationship feature between the two joint points, that is, the joint pair relationship feature, so as to obtain the joint pair relationship feature set corresponding to each joint pair category:

[0156]

[0157] in, Used to represent the relationship features of joint pairs, corresponding to v in formula (1); Joint point features used to represent a joint category; Joint point features used to represent another joint class.

[0158] Optionally, in the above Figure 2 Based on the corresponding embodiment, in another optional embodiment of the multi-person posture estimation method provided in the embodiment of the present application, as Figure 6 As shown, step S104 samples the joint pair relationship feature set based on the sampling strategy to obtain reference joint pair features, reference joint pair positive sample features and reference joint pair negative sample features, including:

[0159] In step S601, any one of the joint pair relationship features in the joint pair relationship feature set is used as a reference joint pair feature;

[0160] In step S602, the joint pair relationship features other than the reference joint pair features in the joint pair relationship feature set are used as reference joint pair positive sample features;

[0161] In step S603, a reference joint pair negative sample feature corresponding to the reference joint pair feature is determined from other joint pair relationship feature sets.

[0162] In this embodiment, after obtaining the joint pair relationship feature set, based on a preset difficult sampling strategy, any joint pair relationship feature in the joint pair relationship feature set can be used as a reference joint pair feature, and the joint pair relationship features in the joint pair relationship feature set other than the reference joint pair feature can be used as reference joint pair positive sample features. Then, the reference joint pair negative sample features corresponding to the reference joint pair features can be determined from other joint pair relationship feature sets, which can achieve data enhancement and further enhance the joint expression, so that the relationship between joint features of different human body categories between images can be explored, so that more robust joint features can be better learned.

[0163] Specifically, after obtaining the joint pair relationship feature set, the joint pair relationship feature set can be sampled based on a preset difficult sampling strategy. Specifically, any joint pair relationship feature in the joint pair relationship feature set can be used as a reference joint pair feature, such as the joint pair feature composed of the elbow joint and wrist joint of the arm. Furthermore, the joint pair relationship features in the joint pair relationship feature set except the reference joint pair feature can be used as reference joint pair positive sample features. Then, the first feature similarity between the reference joint pair feature and the joint pair relationship features in other joint pair relationship feature sets can be calculated, and based on the first feature similarity, the reference joint pair negative sample features can be screened out from other joint pair relationship feature sets.

[0164] It should be noted that for the design of sampling strategies, other construction methods can be tried for the selection of reference joint pairs and negative samples in pairwise relationship constraints. For example, joint pairs can construct more pairs between joints as a group for constraint construction, so as to explore better positioning performance.

[0165] Optionally, in the above Figure 6 Based on the corresponding embodiment, in another optional embodiment of the multi-person posture estimation method provided in the embodiment of the present application, as Figure 7 As shown, step S603 determines the reference joint pair negative sample features corresponding to the reference joint pair features from other joint pair relationship feature sets, including:

[0166] In step S701, a first feature similarity between a reference joint pair feature and joint pair relationship features in other joint pair relationship feature sets is calculated;

[0167] In step S702, based on the first feature similarity, the reference joint pair negative sample features are determined.

[0168] In this embodiment, after obtaining the reference joint pair features, in order to better make the representations of joint pairs of the same category more consistent and the representations of joint pairs of different categories more dissimilar, thereby learning a better expression of the structural relationship between joints, this embodiment can calculate the first feature similarity between the reference joint pair features and the joint pair relationship features in other joint pair relationship feature sets, and then, based on the first feature similarity, the reference joint pair negative sample features can be filtered out from other joint pair relationship feature sets.

[0169] Specifically, after obtaining the reference joint pair features, the similarity between the reference joint pair features and the joint pair relationship features in other joint pair relationship feature sets can be calculated according to a similarity algorithm such as a distance calculation formula, a cosine similarity calculation formula, or other similarity calculation formulas, which are not specifically limited here, that is, the first feature similarity. Then, they can be sorted according to the first feature similarity, and the top-H joint pair relationship features with the highest similarity can be taken as the reference joint pair negative sample features, where H is a hyperparameter set according to actual application requirements, which is not specifically limited here, and is usually H = 4.

[0170] Optionally, in the above Figure 3 Based on the corresponding embodiment, in another optional embodiment of the multi-person posture estimation method provided in the embodiment of the present application, as Figure 8 As shown, step S302 samples the category joint feature set based on the sampling strategy to obtain reference joint point features, reference joint point positive sample features and reference joint point negative sample features, including:

[0171] In step S801, from the category joint feature set, reference joint point features whose predicted loss values ​​meet the screening conditions are determined;

[0172] In step S802, the joint point features other than the reference joint point features in the category joint feature set are used as the reference joint point positive sample features;

[0173] In step S803, a reference joint point negative sample feature corresponding to the reference joint point feature is determined from a set of joint features of other categories.

[0174] In this embodiment, after obtaining the category joint feature set, based on the preset difficulty sampling strategy, the reference joint point features whose predicted loss values ​​meet the screening conditions can be screened out from the category joint feature set, and the joint point features other than the reference joint point features in the category joint feature set are used as the reference joint point positive sample features. Then, the reference joint point negative sample features corresponding to the reference joint point features are screened out from other category joint feature sets, which can achieve data enhancement and further enhance the joint expression, so that the relationship between the joint features of different human categories between images can be explored, so that more robust joint features can be better learned.

[0175] Specifically, after obtaining the category joint feature set, the category joint feature set can be sampled based on a preset difficulty sampling strategy. The reference joint point features whose predicted loss values ​​meet the screening conditions can be first screened out from the category joint feature set. Specifically, the reference joint points can be selected based on the reliability of the joint prediction results, that is, when the predicted loss value of a joint is relatively large, the joint point feature corresponding to the predicted loss value greater than the preset loss threshold is selected as the reference joint point (i), or it can be selected based on the uncertainty of the prediction, where the vector of the joint point feature is expressed as (C×H×W), C is the number of channels, H is the feature height, and W is the feature width, to indicate the certainty of the prediction. When the uncertainty of the prediction of a joint is relatively large, it can be considered to be a more challenging joint point feature, that is, the reference joint point (i). The reference joint point features can also be selected in other ways, and there is no specific restriction here.

[0176] Furthermore, the joint point features other than the reference joint point features in the category joint feature set can be used as the reference joint point positive sample features. Then, the second feature similarity between the reference joint point features and the joint point features in other category joint feature sets can be calculated, and based on the second feature similarity, the reference joint point negative sample features can be filtered out from other category joint feature sets.

[0177] It should be noted that, for the design of the sampling strategy, for the features of a single joint, you can also try to cluster the features of each class, use the class center instead of calculation to obtain the reference joint point, or use other sampling methods, which are not specifically limited here.

[0178] Optionally, in the above Figure 8 Based on the corresponding embodiment, in another optional embodiment of the multi-person posture estimation method provided in the embodiment of the present application, as Fig. 9 As shown, step S803 determines the reference joint point negative sample features corresponding to the reference joint point features from the joint feature sets of other categories, including:

[0179] In step S901, the second feature similarity between the reference joint point feature and the joint point features in the joint feature set of other categories is calculated;

[0180] In step S902, the negative sample features of the reference joint points are determined according to the second feature similarity.

[0181] In this embodiment, after obtaining the reference joint point features, in order to better make the joint features of the same category between different people more consistent and the joint features of different categories more dispersed, thereby learning better joint features, this embodiment can calculate the second feature similarity between the reference joint point features and the joint point features in other categories of joint feature sets, and then, based on the second feature similarity, the reference joint point negative sample features can be filtered out from the joint feature sets of other categories.

[0182] Specifically, after obtaining the reference joint point features, the similarity between the reference joint point features and the joint point features in other categories of joint feature sets can be calculated according to a similarity algorithm such as a distance calculation formula, a cosine similarity calculation formula, or other similarity calculation formulas, which are not specifically limited here, that is, the second feature similarity. Then, they can be sorted according to the second feature similarity, and the top-H joint pair relationship features with the highest similarity can be taken as the reference joint point negative sample features, where H is a hyperparameter set according to actual application requirements, which is not specifically limited here, and is usually H = 6.

[0183] Optionally, in the above Figure 2 Based on the corresponding embodiment, in another optional embodiment of the multi-person posture estimation method provided in the embodiment of the present application, as Fig.10 As shown, step S102 calculates the position loss of each joint point feature in the joint point feature set based on the data annotation to obtain the predicted loss value corresponding to each joint point feature, including:

[0184] In step S1001, each joint point feature in the joint point feature set is decoded through the decoding structure of the posture estimation model to obtain the corresponding predicted position of each joint point feature in the sample image;

[0185] In step S1002, the position loss is calculated for the predicted position based on the data annotation to obtain the predicted loss value corresponding to each joint point feature.

[0186] In this embodiment, after obtaining the joint point feature set, each joint point feature in the joint point feature set can be decoded through the decoding structure of the posture estimation model to obtain the corresponding predicted position of each joint point feature in the sample image, and then, the position loss is calculated for the predicted position using the data annotation information carried by each sample image to obtain the predicted loss value corresponding to each joint point feature.

[0187] Specifically, Fig.13 As shown, after obtaining the joint feature set, the decoding structure ψ of posture estimation can be used r, get the predicted position corresponding to each joint point feature, then use groundtruth to constrain the output predicted position, the constraint can be classification constraint, regression constraint, or other constraint methods, no specific restrictions are made here, that is, position loss calculation is performed, which can be classification loss calculation, position loss calculation, or other loss calculation methods, no specific restrictions are made here, to obtain the predicted loss value corresponding to each joint point feature For example, since the data of the model function can usually appear in the form of (x, y), where x is the previous input data and y is the value predicted by the model, that is, the predicted position. The data annotation will be compared with the result predicted by the model, that is, the predicted position. For example, y and t can be compared in the loss function (loss function / error function) of the following formula (4) to calculate the loss (loss / error), that is, the predicted loss value, such as in the minimum variance:

[0188]

[0189] Among them, L is used to represent the prediction loss value, and m is used to represent the number of prediction positions.

[0190] Optionally, in the above Figure 2 Based on the corresponding embodiment, in another optional embodiment of the multi-person posture estimation method provided in the embodiment of the present application, as Fig.11 As shown, step S101 extracts a joint point feature set corresponding to each target object in a plurality of target objects from each sample image based on a posture estimation model, including:

[0191] In step S1101, each sample image is detected by the target detector of the posture estimation model to obtain the score and position corresponding to each target object;

[0192] In step S1102, based on the score and position corresponding to each target object, the feature extractor of the posture estimation model extracts joint features of each target object to obtain a joint point feature set.

[0193] In this embodiment, after obtaining several batches of sample images, the object recognition can be performed on each sample image in each batch through the target detector of the pose estimation model on the basis of the framework of the conventional pose estimation model to obtain the score and position corresponding to each target object. Then, based on the score and position corresponding to each target object, the feature extractor of the pose estimation model can be used to extract joint features of each target object to extract a set of joint point features corresponding to each target object in multiple target objects from each sample image.

[0194] The target detector may be a Faster-RCNN network or a Mask-RCNN network, or other target detectors, which are not specifically limited here. The sample image may be specifically an image containing at least one whole body of a human body, or an image of a part of a human body, which are not specifically limited here. The feature extractor is a convolutional neural network that can be used for feature extraction.

[0195] Specifically, Fig.13 As shown, ResNet and HRNet can be used as the posture estimation model of the backbone network, and a two-stage processing is performed on each sample image in each batch based on a top-down method. For example, in the first stage, the target detector in the posture estimation model, that is, the human body detector, can be used to perform object detection on each sample image in a batch to obtain the score and position corresponding to each target object, thereby obtaining a collection of different human bodies, that is, multiple target objects. Specifically, a sample image can be input into the target detector, and the convolutional neural network corresponding to the target detector is passed through to output the score of each human body in the sample image and the position of each human body on the sample image, such as (x min ,y min ,x max ,y max , score), where (x min ,y min ) and (x max ,y max ) are used to indicate the upper left corner and lower right corner coordinates of the human body frame, i.e., the target object detection frame. Then, the target object detection frame can be cropped, i.e., a set of human body pictures (a set of different human bodies) can be obtained.

[0196] Furthermore, feature extraction is performed through a feature extractor to extract multiple joint point features corresponding to each target object, and the joint point feature set is obtained by integrating them. Specifically, the feature extractor can extract and compress high-level features of the score and position corresponding to each target object. For example, an image in a human body picture set obtained by cropping the target object detection frame can be input into the feature extractor. After passing through the convolutional neural network structure of the feature extractor, the output feature vector dimension is (C×H×W), where C is the number of channels, H is the feature height, and W is the feature width, that is, the joint point feature, so that the joint point feature set can be summarized.

[0197] Optionally, in the above Figure 2 Based on the corresponding embodiment, in another optional embodiment of the multi-person posture estimation method provided in the embodiment of the present application, as Fig.12As shown, after step S106 adjusts the parameters of the posture estimation model based on the pairwise consistency loss value and the prediction loss value to obtain the target posture estimation model, the method further includes:

[0198] In step S1201, when a posture estimation request is received, an image to be processed is acquired according to the posture estimation request;

[0199] In step S1202, the image to be processed is input into a target posture estimation model, and the target posture estimation model outputs a posture estimation result corresponding to each target object to obtain human body posture estimation data corresponding to the image to be processed.

[0200] In this embodiment, since the image target detection algorithm based on the posture estimation model is an end-to-end algorithm, the customer data can be directly accessed after the target posture estimation model is trained. Furthermore, when a posture estimation request is received, the image to be processed can be obtained according to the posture estimation request, and the image to be processed can be input into the target posture estimation model. The posture estimation result corresponding to each target object is output through the target posture estimation model to obtain the human body posture estimation data corresponding to the image to be processed, so that the obtained human body posture estimation data can be used as a reference in the target application scenario.

[0201] Specifically, after the target posture estimation model is trained, it can be connected to customer data for use. When a posture estimation request is received, the image to be processed can be downloaded or obtained from the database or big data platform according to the image identifier carried in the posture estimation request. Then, the image to be processed can be input into the trained target posture estimation model. The target posture estimation model is used to perform posture estimation on the target object contained in the image to be processed, and the posture estimation result corresponding to each target object is output. In this way, the human posture estimation data corresponding to the image to be processed can be sorted out and applied to many target application scenarios such as film animation, virtual reality, autonomous driving, video detection, motion analysis, etc.

[0202] The following is a detailed description of the multi-person posture estimation device in this application. Fig.15 , Fig.15 This is a schematic diagram of an embodiment of a multi-person posture estimation device in an embodiment of the present application. The multi-person posture estimation device 20 includes:

[0203] An acquisition unit 201 is used to extract a joint point feature set corresponding to each target object in a plurality of target objects from each sample image based on a posture estimation model;

[0204] The processing unit 202 is used to calculate the position loss of each joint point feature in the joint point feature set based on the data annotation to obtain a predicted loss value corresponding to each joint point feature;

[0205] The processing unit 202 is further used to classify the joint point feature set into joint pairs based on the human body joint association relationship, and obtain a joint pair relationship feature set corresponding to each joint pair category;

[0206] A determination unit 203 is used to sample the joint pair relationship feature set based on a sampling strategy to obtain reference joint pair features, reference joint pair positive sample features, and reference joint pair negative sample features;

[0207] The processing unit 202 is further used to calculate the consistency loss based on the reference joint pair features, the reference joint pair positive sample features and the reference joint pair negative sample features to obtain a pairwise consistency loss value corresponding to each joint pair category;

[0208] The processing unit 202 is further used to adjust the parameters of the posture estimation model based on the pairwise consistency loss value and the prediction loss value to obtain a target posture estimation model.

[0209] Optionally, in the above Fig.15 On the basis of the corresponding embodiment, in another embodiment of the multi-person posture estimation device provided by the embodiment of the present application,

[0210] The processing unit 202 is further used to classify the joint point feature set into joint categories based on the joint category labels to obtain a category joint feature set corresponding to each joint category;

[0211] The determination unit 203 is further used to sample the category joint feature set based on the sampling strategy to obtain reference joint point features, reference joint point positive sample features and reference joint point negative sample features;

[0212] The processing unit 202 is further used to calculate the consistency loss based on the reference joint point features, the reference joint point positive sample features and the reference joint point negative sample features to obtain a single joint consistency loss value corresponding to each joint category;

[0213] The processing unit 202 can be specifically used to: adjust the parameters of the posture estimation model based on the single-joint consistency loss value, the pairwise consistency loss value and the prediction loss value to obtain the target posture estimation model.

[0214] Optionally, in the above Fig.15 On the basis of the corresponding embodiment, in another embodiment of the multi-person posture estimation device provided by the embodiment of the present application,

[0215] The processing unit 202 is further used to map each joint point feature in all joint point feature sets to the target space;

[0216] The processing unit 202 is further used to perform dimensionality reduction processing on each joint point feature in the target space to obtain a target dimension joint feature set;

[0217] The processing unit 202 can be specifically used to: divide the target dimension joint feature set into categories of joint pairs based on the human body joint association relationship, and obtain a joint pair relationship feature set corresponding to each joint pair category;

[0218] The processing unit 202 can be specifically used to: divide the target dimension joint feature set into joint categories based on the joint category labels, and obtain a category joint feature set corresponding to each joint category.

[0219] Optionally, in the above Fig.15 On the basis of the corresponding embodiment, in another embodiment of the multi-person posture estimation device provided by the embodiment of the present application, the processing unit 202 can be specifically used for:

[0220] From the target dimension joint feature set, determine the pairwise adjacent joint point features that satisfy the human joint association relationship, where each pairwise adjacent joint point feature corresponds to a joint pair category;

[0221] The joint pair relationship features are calculated based on the features of the two adjacent joint points to obtain the joint pair relationship feature set corresponding to each joint pair category.

[0222] Optionally, in the above Fig.15 On the basis of the corresponding embodiment, in another embodiment of the multi-person posture estimation device provided by the embodiment of the present application, the determination unit 203 can be specifically used for:

[0223] Taking any joint pair relationship feature in the joint pair relationship feature set as a reference joint pair feature;

[0224] The joint pair relationship features other than the reference joint pair features in the joint pair relationship feature set are used as the reference joint pair positive sample features;

[0225] From the set of other joint pair relationship features, a reference joint pair negative sample feature corresponding to the reference joint pair feature is determined.

[0226] Optionally, in the above Fig.15 On the basis of the corresponding embodiment, in another embodiment of the multi-person posture estimation device provided by the embodiment of the present application, the determination unit 203 can be specifically used for:

[0227] Calculate the first feature similarity between the reference joint pair feature and the joint pair relationship features in other joint pair relationship feature sets;

[0228] According to the first feature similarity, the reference joint pair negative sample features are determined.

[0229] Optionally, in the above Fig.15 On the basis of the corresponding embodiment, in another embodiment of the multi-person posture estimation device provided by the embodiment of the present application, the determination unit 203 can be specifically used for:

[0230] From the category joint feature set, determine the reference joint point features whose predicted loss values ​​meet the screening conditions;

[0231] The joint point features other than the reference joint point features in the category joint feature set are used as the reference joint point positive sample features;

[0232] From the set of joint features of other categories, determine the negative sample features of the reference joint points corresponding to the reference joint point features.

[0233] Optionally, in the above Fig.15 On the basis of the corresponding embodiment, in another embodiment of the multi-person posture estimation device provided by the embodiment of the present application, the determination unit 203 can be specifically used for:

[0234] Calculate the second feature similarity between the reference joint point feature and the joint point features in the joint feature set of other categories;

[0235] According to the second feature similarity, the negative sample feature of the reference joint point is determined.

[0236] Optionally, in the above Fig.15 On the basis of the corresponding embodiment, in another embodiment of the multi-person posture estimation device provided by the embodiment of the present application, the processing unit 202 can be specifically used for:

[0237] Through the decoding structure of the posture estimation model, each joint point feature in the joint point feature set is decoded to obtain the corresponding predicted position of each joint point feature in the sample image;

[0238] The position loss is calculated for the predicted position based on the data annotation to obtain the predicted loss value corresponding to each joint point feature.

[0239] Optionally, in the above Fig.15 On the basis of the corresponding embodiment, in another embodiment of the multi-person posture estimation device provided by the embodiment of the present application, the acquisition unit 201 can be specifically used for:

[0240] Each sample image is detected by the target detector of the pose estimation model to obtain the score and position of each target object;

[0241] Based on the score and position corresponding to each target object, the feature extractor of the posture estimation model extracts joint features of each target object to obtain a set of joint point features.

[0242] Optionally, in the above Fig.15 On the basis of the corresponding embodiment, in another embodiment of the multi-person posture estimation device provided by the embodiment of the present application,

[0243] The acquisition unit 201 is further configured to acquire the image to be processed according to the posture estimation request when a posture estimation request is received;

[0244] The processing unit 202 is further used to input the image to be processed into the target posture estimation model, and output the posture estimation result corresponding to each target object through the target posture estimation model to obtain the human body posture estimation data corresponding to the image to be processed.

[0245] On the other hand, the present application provides another schematic diagram of a computer device, such as Fig.16 As shown, Fig.16 3 is a schematic diagram of a computer device structure provided in an embodiment of the present application. The computer device 300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 310 (for example, one or more processors) and a memory 320, and one or more storage media 330 (for example, one or more mass storage devices) storing application programs 331 or data 332. Among them, the memory 320 and the storage medium 330 may be temporary storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the computer device 300. Furthermore, the central processing unit 310 may be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the computer device 300.

[0246] The computer device 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input and output interfaces 360, and / or one or more operating systems 333, such as Windows Server 2000, Windows Server 2003, Windows Server 2003E, Windows Server 2003R ... TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0247] The computer device 300 is also used to execute the following steps: Figures 2 to 12 The steps in the corresponding embodiments.

[0248] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following Figures 2 to 12The illustrated embodiments describe steps in a method.

[0249] Another aspect of the present application provides a computer program product comprising a computer program, which, when executed by a processor, implements the following Figures 2 to 12 The illustrated embodiments describe steps in a method.

[0250] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0251] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0252] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0253] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0254] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

Claims

1. A multi-person posture estimation method, characterized in that: include: Based on the posture estimation model, a joint point feature set corresponding to each target object in multiple target objects is extracted from each sample image; Based on the data annotation, a position loss calculation is performed on each joint point feature in the joint point feature set to obtain a predicted loss value corresponding to each joint point feature; Based on the human body joint association relationship, the joint point feature set is divided into categories of joint pairs to obtain a joint pair relationship feature set corresponding to each joint pair category; Based on the sampling strategy, the joint pair relationship feature set is sampled to obtain reference joint pair features, reference joint pair positive sample features, and reference joint pair negative sample features; Based on the reference joint pair features, the reference joint pair positive sample features and the reference joint pair negative sample features, a consistency loss calculation is performed to obtain a pairwise consistency loss value corresponding to each of the joint pair categories; The parameters of the pose estimation model are adjusted based on the pairwise consistency loss value and the predicted loss value to obtain a target pose estimation model.

2. The method according to claim 1, characterized in that Before adjusting the parameters of the pose estimation model based on the pairwise consistency loss value and the predicted loss value to obtain the target pose estimation model, the method further includes: Based on the joint category labels, the joint point feature set is divided into joint categories to obtain a category joint feature set corresponding to each joint category; Based on the sampling strategy, the category joint feature set is sampled to obtain reference joint point features, reference joint point positive sample features, and reference joint point negative sample features; Based on the reference joint point features, the reference joint point positive sample features and the reference joint point negative sample features, a consistency loss calculation is performed to obtain a single joint consistency loss value corresponding to each of the joint categories; The step of adjusting parameters of the attitude estimation model based on the pairwise consistency loss value and the predicted loss value to obtain the target attitude estimation model includes: The parameters of the posture estimation model are adjusted based on the single-joint consistency loss value, the pairwise consistency loss value, and the predicted loss value to obtain the target posture estimation model.

3. The method according to claim 2, characterized in that Before dividing the joint point feature set into categories of joint pairs based on the human body joint association relationship to obtain the joint pair relationship feature set corresponding to each joint pair category, the method further includes: Mapping each joint point feature in all the joint point feature sets to the target space; Performing dimensionality reduction processing on each joint point feature in the target space to obtain a target dimension joint feature set; Based on the human body joint association relationship, the joint point feature set is divided into categories of joint pairs to obtain a joint pair relationship feature set corresponding to each joint pair category, including: Based on the human joint association relationship, the target dimension joint feature set is divided into categories of joint pairs to obtain a joint pair relationship feature set corresponding to each joint pair category; Based on the joint category labels, the joint point feature set is divided into joint categories to obtain a category joint feature set corresponding to each joint category, including: Based on the joint category labels, the target dimension joint feature set is divided into joint categories to obtain a category joint feature set corresponding to each joint category.

4. The method according to claim 3, characterized in that: Based on the human joint association relationship, the target dimension joint feature set is divided into categories of joint pairs to obtain a joint pair relationship feature set corresponding to each joint pair category, including: Determine, from the target dimension joint feature set, pairwise adjacent joint point features that satisfy the human body joint association relationship, wherein each pairwise adjacent joint point feature corresponds to one of the joint pair categories; The joint pair relationship features are calculated based on the pairwise adjacent joint point features to obtain a joint pair relationship feature set corresponding to each joint pair category.

5. The method according to claim 1, characterized in that: The method of sampling the joint pair relationship feature set based on the sampling strategy to obtain reference joint pair features, reference joint pair positive sample features and reference joint pair negative sample features includes: Using any one joint pair relationship feature in the joint pair relationship feature set as the reference joint pair feature; Using the joint pair relationship features other than the reference joint pair features in the joint pair relationship feature set as the reference joint pair positive sample features; The reference joint pair negative sample feature corresponding to the reference joint pair feature is determined from the other joint pair relationship feature sets.

6. The method according to claim 5, characterized in that The step of determining the reference joint pair negative sample feature corresponding to the reference joint pair feature from the other joint pair relationship feature sets includes: Calculate the first feature similarity between the reference joint pair feature and the joint pair relationship features in the other joint pair relationship feature sets; Determine the negative sample feature of the reference joint pair according to the first feature similarity.

7. The method according to claim 2, characterized in that The method of sampling the category joint feature set based on the sampling strategy to obtain reference joint point features, reference joint point positive sample features, and reference joint point negative sample features includes: Determining, from the category joint feature set, the reference joint point feature whose predicted loss value satisfies a screening condition; Using the joint point features other than the reference joint point features in the category joint feature set as the reference joint point positive sample features; The reference joint point negative sample feature corresponding to the reference joint point feature is determined from the other category joint feature sets.

8. The method according to claim 7, characterized in that The step of determining the reference joint point negative sample feature corresponding to the reference joint point feature from a set of joint features of other categories includes: Calculate the second feature similarity between the reference joint point feature and the joint point features in the other category joint feature sets; Determine the reference joint point negative sample feature according to the second feature similarity.

9. The method according to claim 1, characterized in that: The step of performing position loss calculation on each joint point feature in the joint point feature set based on the data annotation to obtain a predicted loss value corresponding to each joint point feature includes: Decoding each joint point feature in the joint point feature set through the decoding structure of the posture estimation model to obtain a corresponding predicted position of each joint point feature in the sample image; The position loss is calculated for the predicted position based on the data annotation to obtain a predicted loss value corresponding to each joint point feature.

10. The method according to claim 1, characterized in that The method of extracting a joint point feature set corresponding to each target object in a plurality of target objects from each sample image based on the posture estimation model includes: Detecting each of the sample images using the target detector of the posture estimation model to obtain a score and a position corresponding to each of the target objects; Based on the score and position corresponding to each of the target objects, joint features are extracted for each of the target objects through the feature extractor of the posture estimation model to obtain the joint point feature set.

11. The method according to claim 1, characterized in that: After adjusting the parameters of the posture estimation model based on the pairwise consistency loss value and the predicted loss value to obtain the target posture estimation model, the method further includes: When a posture estimation request is received, acquiring an image to be processed according to the posture estimation request; The image to be processed is input into the target posture estimation model, and the posture estimation result corresponding to each target object is output through the target posture estimation model to obtain the human body posture estimation data corresponding to the image to be processed.

12. A multi-person posture estimation device, characterized in that: include: An acquisition unit, used for extracting a joint point feature set corresponding to each target object in a plurality of target objects from each sample image based on a posture estimation model; A processing unit, configured to perform position loss calculation on each joint point feature in the joint point feature set based on data annotation, to obtain a predicted loss value corresponding to each joint point feature; The processing unit is further used to classify the joint point feature set into joint pairs based on the human body joint association relationship, so as to obtain a joint pair relationship feature set corresponding to each joint pair category; A determination unit, configured to sample the joint pair relationship feature set based on a sampling strategy to obtain reference joint pair features, reference joint pair positive sample features, and reference joint pair negative sample features; The processing unit is further used to calculate the consistency loss based on the reference joint pair features, the reference joint pair positive sample features and the reference joint pair negative sample features to obtain a pairwise consistency loss value corresponding to each of the joint pair categories; The processing unit is further used to adjust parameters of the posture estimation model based on the paired consistency loss value and the predicted loss value to obtain a target posture estimation model.

13. A computer device comprising a memory, a processor and a bus system, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images