A multi-person pose estimation method, system and device

Abstract

This invention discloses a method, system, and device for multi-person pose estimation, comprising: inputting a feature map F in As input to the pre-constructed occluder branch, the occluder feature map F is obtained. out1 And perform key point localization to obtain the key point heatmap of the occluder; input feature map F in Feature map F of the occluder out1 The input feature map F is obtained by fusion. in_occludee ; Input feature map F in_occludee As input to the pre-constructed occluded branch, obtain the occluded feature map F. out2 And perform key point localization to obtain the key point heatmap H of the occluded subject. occludee The output is the pose estimation result of multiple people in the image to be estimated. This invention realizes the display modeling of complex scenes with crowds or occlusions and decouples the occlusion relationship, and establishes different information spaces for occluders and occluded people. It avoids missed or false detection of key points and effectively improves the estimation accuracy.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a multi-person pose estimation method, system and device. BACKGROUND

[0002] In computer vision analysis, human pose is an important reference for computer analysis of human behavior; human pose structure contains joint topological structure information, and is a simple and rich human behavior representation method; therefore, human pose estimation is widely studied as a basic task and applied to fields such as behavior recognition, automatic driving, virtual reality and augmented reality.

[0003] Multi-person pose estimation is a technology for obtaining all persons in a video image and positioning key points for each person; in the prior art, a multi-person pose estimation method based on top-down first detects a human bounding box using a target detection method, and crops a human image from the original image, then sequentially uses a single-person pose estimation method thereon, and finally obtains a multi-person pose result by summarizing multiple single-person poses; compared with a bottom-up-based pose estimation method, the method based on top-down often shows higher estimation performance.

[0004] However, in the multi-person pose estimation method based on top-down, there is a premise assumption that there is only one person in a single bounding box, which shows excellent estimation performance in most simple ideal scenes; however, for complex scenes such as crowded or occluded scenes that commonly exist in reality, multiple persons often appear in a single human bounding box, and the above premise assumption does not hold; in the face of crowded or occluded complex scenes, the existing multi-person pose estimation method based on top-down can only estimate and identify the single-person pose in the above complex scenes, resulting in problems such as missing detection or false detection of part of the key points, which greatly reduces the accuracy of the pose estimation result. SUMMARY

[0005] In view of the technical problems in the prior art, the present application provides a multi-person pose estimation method, system and device to solve the problem that the existing multi-person pose estimation method based on top-down has missing detection or false detection of part of the key points when facing complex scenes such as crowded or occluded scenes, which greatly reduces the accuracy of the pose estimation result.

[0006] To achieve the above object, the technical scheme adopted by the present application is as follows:

[0007] The present application provides a multi-person pose estimation method, comprising the following steps:

[0008] performing feature extraction on the image to be estimated to obtain an input feature map F in ;

[0009] The input feature map F in As input to the pre-constructed occluder branch ORB, the occluder feature map F in the image to be estimated is obtained. out1 ; and using a preset first pose prediction head to estimate the occluder feature map F in the image to be estimated. out1 The key points are located to obtain the key point heatmap H of the occluder. occluder ;

[0010] The input feature map F in With the feature map F of the occluder out1 Feature fusion is performed to obtain the input feature map F. in_occludee ;

[0011] The input feature map F in_occludee As input to the pre-constructed Occluded Part of the Edge (OEB) branch, the occluded feature map F in the image to be estimated is obtained. out2 ; and using a preset second pose prediction head to estimate the occluded feature map F in the image to be estimated. out2 Keypoint localization was performed to obtain the keypoint heatmap H of the occluded subject. occludee ;

[0012] The key point heatmap H of the obstructor occluder and the key point heatmap H of the obscured person occludee As output, the multi-person pose estimation result of the image to be estimated is obtained.

[0013] Furthermore, feature extraction is performed on the image to be estimated to obtain the feature map F. in The process is as follows:

[0014] Using a pre-selected backbone network, features are extracted from the image to be estimated to obtain the feature map F. in The pre-selected backbone network is an HRNet feature extractor, a ResNet feature extractor, or a SimpleBaseline feature extractor.

[0015] Furthermore, the pre-constructed occluder branch ORB includes a first attention module, a first convolution module, a second attention module, and a second convolution module connected in sequence.

[0016] The first attention module is used to process the input feature map F in Attention mechanism computation is performed to obtain the occluder feature map F. attention1 ;

[0017] The first convolutional module is used to process the occluder feature map F attention1 Perform two convolution operations to obtain the occluder feature map F′. in1 ;

[0018] The second attention module is configured to perform attention mechanism operation on the occluder feature map F in1 to obtain an occluder feature map F attention1 .

[0019] The second convolution module is configured to perform twice convolution operation on the occluder feature map F attention1 to obtain an occluder feature map F in the image to be estimated. out1 .

[0020] Further, the first attention module and the second attention module have the same structure, and both include a channel attention module and a spatial attention module.

[0021] The channel attention module is configured to increase the weight of a preset channel, and the spatial attention module is configured to increase the weight required by a space to be focused.

[0022] Further, the key point heat map of the occluder is H occluder .

[0023] H occluder =Conv pred (F out1 ).

[0024] Further, the pre-constructed occludee branch OEB includes a third attention module, a third convolution module, a fourth attention module and a fourth convolution module connected in sequence.

[0025] The third attention module is configured to perform attention mechanism operation on the input feature map F in_occludee to obtain an occludee feature map F attention2 .

[0026] The third convolution module is configured to perform twice convolution operation on the occluder feature map F attention2 to obtain an occludee feature map F in2 .

[0027] The fourth attention module is configured to perform attention mechanism operation on the occluder feature map F in2 to obtain an occludee feature map F attention2 .

[0028] The second convolution module is configured to perform twice convolution operation on the occluder feature map F attention2 to obtain an occludee feature map F in the image to be estimated. out2 .

[0029] Further, the third attention module and the fourth attention module are the same in structure, and each includes a channel attention module and a spatial attention module.

[0030] The channel attention module is configured to increase the weight of a preset channel, and the spatial attention module is configured to increase the weight required by a space of interest.

[0031] Further, the key point heat map H occludee of the occluded person is:

[0032] H occludee = Conv pred (F out2 ).

[0033] The application further provides a multi-person pose estimation system, comprising:

[0034] a feature extraction module configured to perform feature extraction on a to-be-estimated image to obtain an input feature map F in ;

[0035] an occluded person branch network module configured to take the input feature map F in as an input of a pre-constructed occluded person branch ORB, obtain an occluded person feature map F out1 in the to-be-estimated image, and perform key point positioning on the occluded person feature map F out1 in the to-be-estimated image by using a preset first pose prediction head to obtain a key point heat map H occluder of the occluded person;

[0036] an occlusion guiding module configured to perform feature fusion on the input feature map F in and the occluded person feature map F out1 to obtain an input feature map F in_occludee ;

[0037] an occluded person branch network module configured to take the input feature map F in_occludee as an input of a pre-constructed occluded person branch OEB, obtain an occluded person feature map F out2 in the to-be-estimated image, and perform key point positioning on the occluded person feature map F out2 in the to-be-estimated image by using a preset second pose prediction head to obtain a key point heat map H occludee of the occluded person;

[0038] an output module configured to take the key point heat map H occlude- of the occluded person and the key point heat map H occludee of the occluded person as an output to obtain a multi-person pose estimation result of the to-be-estimated image.

[0039] The present invention also provides a multi-person pose estimation device, comprising:

[0040] Memory, used to store computer programs;

[0041] A processor is used to implement the steps of the multi-person pose estimation method when executing the computer program.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0043] This invention provides a method and system for multi-person pose estimation, which utilizes the occluder branch ORB to obtain the occluder feature map F in the image to be estimated. out1 ; Input feature map F in Feature map F of the occluder out1 After feature fusion, the occluded feature map F in the image to be estimated is obtained using the occluded party branch (OEB). out2 By treating the occluder feature map F in the estimated image out1 and the occluded feature map F in the image to be estimated out2 Distributed keypoint localization enables display modeling of complex scenes with crowding or occlusion, decoupling occlusion relationships and establishing separate information spaces for occluders and occluded individuals to estimate the poses of multiple people. From the perspective of the overall dual branches of occluders and occluded individuals, the dual-person keypoint output can well match the labels in crowded scenes, decoupling the complex problem of pose estimation in crowded scenes into two simple problems of predicting the occluders and occluded individuals. It avoids missed or false detections of human keypoints, has low dependence on the quality differences of the generated human bounding boxes, and has high robustness, effectively improving the accuracy of pose estimation results.

[0044] Furthermore, the occluder branch (ORB) and the occluded branch (OEB) utilize convolutional operations to fully model and differentiate the information of both the occluder and the occluded. The integration of attention modules in both branches allows each branch to automatically allocate attention to the same input, learning the characteristic information of the target human pose. This enables clearer modeling of crowded occluded scenes, improving the ability of each branch to learn the features of different people and its reasoning ability in occluded scenarios, thus enhancing the accuracy of the estimation results. Attached Figure Description

[0045] Figure 1 This is a structural diagram of a multi-person pose estimation network model based on crowding-occlusion awareness in the embodiment.

[0046] Figure 2 This is a structural diagram of the crowding-occlusion sensing module in the embodiment;

[0047] Figure 3 The occluder feature map F in the image to be estimated in the embodiment.out1 and the visualization result of the occluder feature map F out2 and the multi-person pose estimation result map; wherein, Figure 3 (a) is the original image, Figure 3 (b) is the visualization result of the occluder feature map F out1 in the image to be estimated, Figure 3 (c) is the visualization result of the occluded feature map F out2 in the image to be estimated; Figure 3 (d) is the multi-person pose estimation result.

[0048] Figure 4 are the pose prediction result maps of six different pose estimation methods. DETAILED DESCRIPTION

[0049] In order to make the technical problems solved by the present application, the technical solutions and beneficial effects clearer, the following specific embodiments are used to further illustrate the present application. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0050] The present application provides a multi-person pose estimation method, comprising the following steps:

[0051] Step 1, feature extraction is performed on the image to be estimated to obtain an input feature map F in ; specifically, a preselected backbone network is used to perform feature extraction on the image to be estimated to obtain the feature map F in ; wherein the preselected backbone network adopts an HRNet feature extractor, a ResNet feature extractor or a SimpleBaseline feature extractor.

[0052] Step 2, the input feature map F in is taken as the input of a pre-constructed occluder branch ORB to obtain an occluder feature map F out1 in the image to be estimated; and a first pose prediction head preset is used to locate the key points of the occluder feature map F out1 in the image to be estimated to obtain an occluder key point heat map H occluder .

[0053] Wherein, the occluder key point heat map is H occluder :

[0054] H occluder =Conv pred (F out1 ).

[0055] The pre-constructed occluder branch ORB in the application comprises a first attention module, a first convolution module, a second attention module and a second convolution module connected in sequence.

[0056] The first attention module is used for performing attention mechanism operation on the input feature map F in to obtain an occluder feature map F attention1 ; the first convolution module is used for performing twice convolution operation on the occluder feature map F attention1 to obtain an occluder feature map F' in1 ; the second attention module is used for performing attention mechanism operation on the occluder feature map F' in1 to obtain an occluder feature map F' attention1 ; and the second convolution module is used for performing twice convolution operation on the occluder feature map F' attention1 to obtain an occluder feature map F out1 in the image to be estimated.

[0057] Step 3, performing feature fusion on the input feature map F in and the occluder feature map F out1 to obtain an input feature map F in_occludee .

[0058] Step 4, taking the input feature map F in_occludee as the input of the pre-constructed occluded person branch OEB, obtaining an occluded person feature map F out2 in the image to be estimated; and performing key point positioning on the occluded person feature map F out2 in the image to be estimated by using a pre-set second pose prediction head to obtain an occluded person key point heat map H occludee .

[0059] The occluded person key point heat map H occludee is as follows:

[0060] H occludee =Conv pred (F out2 ).

[0061] The pre-constructed occluded person branch OEB in the application comprises a third attention module, a third convolution module, a fourth attention module and a fourth convolution module connected in sequence.

[0062] The third attention module is used for performing attention mechanism operation on the input feature map F in_occludee to obtain an occluded person feature map F attention2 ; the third convolution module is used for performing twice convolution operation on the occluded person feature map F attention2Two convolution operations are performed to obtain the occluded person feature map F' in2 The fourth attention module is configured to perform attention mechanism operation on the occluded person feature map F' in2 to obtain the occluded person feature map F' attention2 The second convolution module is configured to perform two convolution operations on the occluded person feature map F' attention2 to obtain the occluded person feature map F out2 in the image to be estimated.

[0063] Step 5, the key point heat map H occluder of the occluded person and the key point heat map H occludee of the occluded person are taken as outputs to obtain the multi-person pose estimation result of the image to be estimated.

[0064] In the present application, the first attention module, the second attention module, the third attention module and the fourth attention module have the same structure and each include a channel attention module and a spatial attention module; the channel attention module is configured to increase the weight of a preset channel; and the spatial attention module is configured to increase the weight required for attention to space.

[0065] The present application also provides a multi-person pose estimation system, which includes a feature extraction module, an occluded person branch network module, an occlusion guiding module, an occluded person branch network module and an output module; the feature extraction module is configured to perform feature extraction on an image to be estimated to obtain an input feature map F in ; the occluded person branch network module is configured to take the input feature map F in as the input of a pre-constructed occluded person branch ORB to obtain an occluded person feature map F out1 in the image to be estimated; and a preset first pose prediction head is used to locate the key points of the occluded person feature map F out1 in the image to be estimated to obtain an occluded person key point heat map H occluder ; the occlusion guiding module is configured to perform feature fusion on the input feature map F in and the occluded person feature map F out1 to obtain an input feature map F in_occludee ; the occluded person branch network module is configured to take the input feature map F in_occludee as the input of a pre-constructed occluded person branch OEB to obtain an occluded person feature map F out2 in the image to be estimated; and a preset second pose prediction head is used to locate the key points of the occluded person feature map F out2 in the image to be estimated to obtain an occluded person key point heat map H occludee ; and the output module is configured to take the occluded person key point heat map H occluderand the key point heat map H of the occluded person occludee As output, the multi-person pose estimation result of the image to be estimated is obtained.

[0066] The application further provides a multi-person pose estimation device, comprising a memory for storing a computer program and a processor for implementing the steps of the multi-person pose estimation method when executing the computer program.

[0067] The processor implements the steps of the multi-person pose estimation method when executing the computer program, for example, performing feature extraction on the image to be estimated to obtain the input feature map F in ; taking the input feature map F in as the input of the pre-constructed occluder branch ORB to obtain the occluder feature map F out1 in the image to be estimated; and using a preset first pose prediction head to perform key point positioning on the occluder feature map F out1 in the image to be estimated to obtain the key point heat map H of the occluded person occluder ; performing feature fusion on the input feature map F in and the occluder feature map F out1 to obtain the input feature map F in_occludee ; taking the input feature map F in_occludee as the input of the pre-constructed occluded person branch OEB to obtain the occluded person feature map F out2 in the image to be estimated; and using a preset second pose prediction head to perform key point positioning on the occluded person feature map F out2 in the image to be estimated to obtain the key point heat map H of the occluded person occludee ; taking the key point heat map H of the occluded person occluder and the key point heat map H of the occluded person occludee as output to obtain the multi-person pose estimation result of the image to be estimated.

[0068] Alternatively, the processor implements the functions of the modules in the above system when executing the computer program, for example, a feature extraction module for performing feature extraction on the image to be estimated to obtain the input feature map F in ; an occluder branch network module for taking the input feature map F in as the input of the pre-constructed occluder branch ORB to obtain the occluder feature map F out1 in the image to be estimated; and using a preset first pose prediction head to perform key point positioning on the occluder feature map F out1 in the image to be estimated to obtain the key point heat map H of the occluded person occluder ; an occlusion guidance module for performing feature fusion on the input feature map F in and the occluder feature map Fout1 perform feature fusion to obtain an input feature map F in_occludee ; an occluder branch network module, configured to take the input feature map F in_occludee as an input of a pre-constructed occluder branch OEB, acquire an occluder feature map F out2 in the image to be estimated, and utilize a preset second pose prediction head to perform key point positioning on the occluder feature map F out2 in the image to be estimated to obtain an occluder key point heat map H occludee ; and an output module, configured to take the occluder key point heat map H occluder and the occluder key point heat map H occludee as outputs to obtain a multi-person pose estimation result of the image to be estimated.

[0069] Exemplarily, the computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a preset function, which are used to describe the execution process of the computer program in the multi-person pose estimation device. For example, the computer program can be divided into a feature extraction module, an occluder branch network module, an occlusion guidance module, an occluded person branch network module, and an output module, and the specific functions of each module are as follows: the feature extraction module is configured to perform feature extraction on an image to be estimated to obtain an input feature map F in ; the occluder branch network module is configured to take the input feature map F in as an input of a pre-constructed occluder branch ORB, acquire an occluder feature map F out1 in the image to be estimated, and utilize a preset first pose prediction head to perform key point positioning on the occluder feature map F out1 in the image to be estimated to obtain an occluder key point heat map H occlude- ; the occlusion guidance module is configured to perform feature fusion on the input feature map F in and the occluder feature map F out1 to obtain an input feature map F in_occludee ; the occluded person branch network module is configured to take the input feature map F in_occludee as an input of a pre-constructed occluded person branch OEB, acquire an occluded person feature map F out2 in the image to be estimated, and utilize a preset second pose prediction head to perform key point positioning on the occluded person feature map F out2 in the image to be estimated to obtain an occluded person key point heat map H occludee ; and the output module is configured to take the occluder key point heat map Hoccluder and the keypoint heat map H of the occluded person occludee As an output, a multi-person pose estimation result of the image to be estimated is obtained.

[0070] The multi-person pose estimation device can be a desktop computer, a notebook computer, a palm computer, a cloud server, or the like. The multi-person pose estimation device can include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the above is an example of the multi-person pose estimation device, and does not constitute a limitation on the multi-person pose estimation device, and can include more components than the above, or combine certain components, or different components, for example, the multi-person pose estimation device can also include an input / output device, a network access device, a bus, and the like.

[0071] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, or the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, and the like. The processor is a control center of the multi-person pose estimation device, and connects various parts of the multi-person pose estimation device through various interfaces and lines.

[0072] The memory can be used to store the computer programs and / or modules, and the processor realizes various functions of the multi-person pose estimation device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory.

[0073] The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required for a function (such as a sound playing function, an image playing function, or the like), and the like; and the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, or the like), and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0074] Example

[0075] This embodiment provides a method for estimating the pose of multiple people, including the following steps:

[0076] Step 1: Obtain the image to be estimated.

[0077] Step 2: Construct a crowd and occlusion-aware network model for multi-person pose estimation (CONet);

[0078] Step 3: Input the image to be estimated into the crowding-occlusion awareness-based multi-person pose estimation network model to obtain the key point heatmap H of the occluder. occluder Key point heatmap H of the obscured person occludee ;

[0079] Step 4: Create a heatmap of the key points of the obstructor H. occluder and the key point heatmap H of the obscured person occludee As output, the multi-person pose estimation result of the image to be estimated is obtained.

[0080] As attached Figure 1 As shown, the multi-person pose estimation network model based on crowding-occlusion awareness includes a backbone network and a crowding-occlusion aware head (COHead); wherein, the backbone network is used to extract features from the image to be estimated to obtain the input feature map F. in The crowding-occlusion perception module will process the input feature map F. in As input, output is the keypoint heatmap H of the occluder. occluder Key point heatmap H of the obscured person occludee This yields the multi-person pose estimation results for the image to be estimated.

[0081] In this embodiment, the backbone network is compatible with any existing feature extractor; for example, HRNet feature extractor, ResNet feature extractor, or SimpleBaseline feature extractor; by using the feature extractor as the backbone network, high-resolution feature maps are maintained at all times while extracting high-level semantic features, so as to ensure the effectiveness of the localization results of the fine-grained information required for key point detection.

[0082] In this embodiment, the crowding-occlusion perception module adopts a dual-branch structure, which is used to display and model crowding or occlusion conditions and decouple occlusion relationships; as shown in the attached figure. Figure 2As shown, the crowded-obstruction perception module includes an occluder branch ORB, a first pose prediction head, an occlusion guidance module, an occludee branch OEB, and a second pose prediction head; wherein the occluder branch ORB is configured to obtain an occluder feature map F in in an image to be estimated according to the input feature map F out1 ; the first pose prediction head is configured to locate key points of the occluder feature map F out1 in the image to be estimated to obtain an occluder key point heat map H occluder ; the occlusion guidance module is configured to fuse the input feature map F in and the occluder feature map F out1 to obtain an input feature map F in_occludee ; the occludee branch OEB is configured to obtain an occludee feature map F in_occludee in the image to be estimated according to the input feature map F out2 ; and the second pose prediction head is configured to locate key points of the occludee feature map F out2 in the image to be estimated to obtain an occludee key point heat map H occludee .

[0083] For crowded scenes, the crowded-obstruction perception module redefines and decomposes the traditional single-person key point detection into a two-person pose estimation task, which is different from the single-person pose estimation that only detects the key points of a single person; wherein the occluder branch ORB is used to model the occluder, and the first pose prediction head is used to predict the key points of the occluder; the occludee branch OEB is used to model the occludee, and the second pose prediction head is used to predict the key points of the occludee; for occlusion scenes, the occluder branch ORB and the occludee branch OEB can decouple the occlusion situation, form two rich information spaces, and more explicitly model the two persons; the crowded-obstruction perception module further includes an occlusion guidance module, which is a residual connection from the occluder branch ORB to the occludee branch OEB. The occlusion guidance module contains rich auxiliary information, such as occlusion relationship, occludee body part, and other high-level semantic information and occlusion area, occlusion point, and other local detail information, thereby guiding the key point detection of the occludee.

[0084] In this embodiment, the design process and principle of the crowded-obstruction perception module are as follows:

[0085] (1) Determination of the number of prediction head branches;

[0086] The number of people in a single bounding box is uncertain, theoretically, the more prediction head branches are used, the more reasonable the crowded scene can be modeled, but too many prediction head branches will increase the model capacity, and will greatly increase the network complexity and inference time; too few prediction head branches will degenerate into a traditional top-down method; since the proportion of the number of human bodies in the human body bounding box in the existing mainstream data set is very small, for example: the proportion of the COCO data set and the CrowdPose data set is 0.1%, and the proportion of the OCHuman data set is 0.9%; and most human body bounding boxes contain 1-2 human bodies, for example: the proportion of the COCO data set and the CrowdPose data set is 99.9%, and the proportion of the OCHuman data set is 99.1%; therefore, in the present application, the number of prediction head branches is set to 2.

[0087] Although the multi-branch multi-prediction head structure can match the supervised information of multiple human key points in the output quantity, the quality of the output key points is questionable, if each prediction head branch can learn the information it needs from the same feature input, it will be very effective for crowded and occluded situations; specifically, the input of the multi-prediction head branch is consistent, all from the features extracted by the backbone network from the original image, and the expected output is the completely different key point information of the occluder and the occluded person; using a simple double-prediction head branch cannot well learn the different features for the same input, that is, it is difficult for the double-branch to learn the different features corresponding to different human bodies; therefore, in the present embodiment, by increasing the network depth of different branches, different people's key point supervision labels are used to guide different branches to learn different features.

[0088] (2) Determine the network depth of the prediction head branch

[0089] In this embodiment, the depth of each prediction head branch is set to 4, that is, 4 layers of convolution; the reason for setting it to 4 layers of convolution is that, on the one hand, considering the model performance and execution delay, if the depth of the prediction head branch is too shallow, the dual prediction head branch cannot fully extract differentiated features, so it is difficult to learn different information corresponding to different poses, such as occlusion relationship, detailed information of occlusion position, etc.; if the depth of the prediction head branch is designed to be too deep, it will directly lead to excessive model capacity and high memory usage, and also cause high network inference delay. On the other hand, considering the effectiveness of the comparison between the dual prediction head branch structure and other model structures, in general, the deeper the network depth, that is, with appropriate overfitting solutions, the better the evaluation performance of the task. In order to avoid this factor verifying the effectiveness of the proposed dual branch structure, that is, the improvement of the evaluation performance of the crowded scene in this embodiment comes from the structure design rather than simply deepening the network depth, this embodiment will try to design the network to be shallower, and 4 layers of convolution is a reasonable depth design, which can verify the effectiveness of the dual branch structure while almost no additional delay overhead.

[0090] In order to enable the dual prediction head branches with the same input to learn different features corresponding to different people, an attention mechanism is used in this embodiment; wherein the attention mechanism enables the model to automatically calculate attention values, so that the model can pay attention to as much information as possible under the condition of limited resources and large amount of information. For the dual branch structure of this embodiment, integrating an attention mechanism for each branch respectively enables them to automatically determine the activation weights of the information corresponding to different people from the same input, and more effectively guides the dual branch to learn differently.

[0091] In this embodiment, the crowded-occlusion perception module includes an occluder branch ORB, a first pose prediction head, an occluded person branch OEB, and a second pose prediction head; wherein the occluder branch ORB and the first pose prediction head are used to detect the key points of the occluder; the occluded person branch OEB and the second pose prediction head are used to detect the key points of the occluded person; wherein the occluder branch includes two attention modules (Convolutional Block Attention Module, CBAM) and two convolution modules; wherein each convolution module includes two convolution layers with a convolution kernel size of 3; a residual connection is arranged between each two convolution layers, which can be used to prevent gradient disappearance and also promote information fusion between individual key points; in this embodiment, in order not to increase the overall operation delay and facilitate verification of the effectiveness of the crowded-occlusion perception module, the channel number of the convolution layer and the channel number of the backbone network are both designed to be 32; the results of the occluded person branch and the occluder branch are exactly the same, which will not be described here.

[0092] In this embodiment, from the perspective of the whole occluder branch ORB and the occluded branch OEB, the two-person key point output can well match the labels in the crowded scene, decoupling the complex problem of crowded scene pose estimation into two simple problems of predicting the occluder and the occluded person; the convolution operation with appropriate depth can fully model and differentiate the information of the occluder and the occluded person; the combination of the attention module in the occluder branch ORB and the occluded branch OEB enables each branch to automatically allocate attention to the same input, learn the feature information of the target human pose, and thus more clearly model the crowded and occluded scene.

[0093] The working process of the occluder branch ORB is described in detail below.

[0094] In this embodiment, the occluder branch ORB includes a first attention module, a first convolution module, a second attention module, and a second convolution module connected in sequence; wherein the first attention module CBAM receives the input feature map F in , and performs attention mechanism operation on the input feature map F in to obtain an occluder feature map F attention1 ; specifically, a channel attention mask A in and a feature map F channel of the input feature map F channel are calculated; a spatial attention mask A channel of the input feature map F spatial is calculated; and the occluder feature map F attention1 is obtained according to the channel attention mask A channel and the spatial attention mask A spatial .

[0095] The occluder feature map F attention1 is:

[0096] F attention1 =(F in ⊙A channel )⊙A spatial .

[0097] In this embodiment, the calculation process of the channel attention mask A channel is as follows:

[0098] The maximum pooling is performed on each channel of the input feature map F in , and then the first activation mask is obtained after full connection operation and sigmoid processing; then, the second activation mask is obtained by using average pooling and full connection operation;

[0099] adding the first activation mask and the second activation mask to obtain a channel attention mask A of size Cx1x1 channel .

[0100] wherein the channel attention mask A channel is:

[0101] A channel = sigmoid(fc(Pool max (F in ))+fc(Pool avg (F in ))).

[0102] In this embodiment, the calculation process of the spatial attention mask A spatial is specifically as follows:

[0103] In order to compress the channel, two ways are used to calculate the spatial attention mask A spatial , one is to find the maximum value along the channel dimension, and the other is to find the average value along the channel dimension; wherein the spatial attention mask A spatial is:

[0104] A spatial = sigmoid(conv(cat(max(F channel ), mean(F channel ))).

[0105] The first convolution module performs twice convolution operation on the occluder feature map F attention1 to obtain an occluder feature map F' in1 ; wherein the occluder feature map F' in1 is:

[0106] F' in1 = Conv2(Conv1(F attention1 )).

[0107] In this embodiment, the working principle and operation process of the second attention module are similar to those of the first attention module; the working principle and operation process of the second convolution module are similar to those of the first convolution module, which will not be described here; in this embodiment, after the occluder feature map F' in1 in the image to be estimated is processed by the second attention module and the second convolution layer, an occluder feature map F out1 in the image to be estimated is obtained.

[0108] In this embodiment, the occluder feature map F out1As input to the first pose prediction head, the occluder feature map F in the image to be estimated is used. out1 The key points are located to obtain the key point heatmap H of the occluder. occluder Among them, the key point heatmap H of the obstructor occluder The size is H×W×N; where N is the number of human key points in the dataset; the key point heatmap H of the occluder. occluder for:

[0109] H occluder =Conv pred (F out1 ).

[0110] In this embodiment, the input to the occluded branch OEB is the input feature map F. in_occludee That is, the input of the occluded branch OEB, in addition to the feature map input feature map F provided by the backbone network, is... in In addition, there is an extra feature map F from the occluder. out1 This additional input is called the occlusion-aware residual connection; wherein, the input feature map F in_occludee The input feature map F is guided by the occlusion module. in With the feature map F of the occluder out1 The elements are added together to obtain the result.

[0111] Aside from the different inputs, the structure and operation flow of the occluded branch OEB and the occluded branch ORB are basically the same, and the specific process will not be described in detail here; that is, the input feature map F in_occludee As input to the pre-constructed Occluded Part of the Edge (OEB) branch, the occluded feature map F in the image to be estimated is obtained. out2 .

[0112] In this embodiment, the occluded feature map F in the image to be estimated is... out2 As input to the second pose prediction head, the occluded feature map F in the image to be estimated is used. out2 Key points are located to obtain the key point heatmap H of the occluded object. occludee .

[0113] Among them, the key point heatmap H of the occluded person occludee The size is H×W×N; where N is the number of human keypoints in the dataset; and where H is the keypoint heatmap of the occluded person. occludee for:

[0114] H occludee =Conv pred (F out2 ).

[0115] The working process of the occlusion guidance module is described below in detail.

[0116] In this embodiment, by designing the occluder branch ORB and the occluded person branch OEB in the crowded-occlusion perception module, the problem of mismatch between the output of the existing top-down method and the supervision information is effectively solved. The double branches in the crowded-occlusion perception module are beneficial to decouple the poses of multiple people in the human bounding box, so as to establish different information spaces to process the poses of different people. However, the multiple people in the same bounding box are often different from two isolated people with a large distance, but are close to each other and have close contact, that is, there is often an interaction between them. The interaction is an effective information. In this embodiment, by designing the occlusion guidance module, the information in the occluder branch ORB and the occluded person branch OEB is fused to make the model learn the information between different people, so as to infer the interaction relationship in subsequent operations.

[0117] Specifically, in this embodiment, the last layer feature of the occluder branch ORB is fused with the input feature of the occluded person branch OEB in an element-wise addition manner; that is, the input feature map F in is fused with the occluder feature map F out1 to obtain the input feature map F in_occludee , which is taken as the input of the occluded person branch OEB. in_occludee

[0118] wherein, the input feature map F in_occludee is:

[0119] F in_occludee = F in +F out1 .

[0120] In this embodiment, the last layer feature of the occluder branch ORB is fused, and the main reason is that the occluder branch ORB is used to detect the occluder, that is, the main person in the target box, so the corresponding feature is more significant. Therefore, taking it as the input for fusion can achieve better results. Otherwise, confusion may occur.

[0121] Experimental results and analysis

[0122] In order to verify the effectiveness of the design of the crowded-occlusion perception based multi-person pose estimation network model in this embodiment on the multi-person pose estimation task, the following verification is carried out on the classic dataset of the multi-person pose estimation task, and the effectiveness of the crowded-occlusion perception based multi-person pose estimation network model is illustrated through the ablation experiment.

[0123] ​In this embodiment, the target detection algorithm is used to process the image to be estimated, and the original image is cropped after obtaining the human body bounding box, and the size of the input image is scaled to 256*192, and the channel number is 3; the input image is used as the input of the multi-person pose estimation network model based on crowdedness and occlusion perception.

[0124] The multi-person pose estimation method described in this embodiment directly outputs the heat map representation method; therefore, the data set needs to be preprocessed before training to generate heat maps corresponding to all key points; wherein each key point coordinate corresponds to a heat map, and the heat map corresponding to each key point is generated using a Gaussian function; secondly, the data set label also needs to be modified; the label format in the traditional pose estimation data set is that each human body bounding box corresponds to a pose label, and the output of the multi-person pose estimation network model based on crowdedness and occlusion perception is a double-person key point result, which does not match the key point label of the traditional data set, so the original data set label needs to be modified to carry out training; specifically, the one-to-one correspondence between the single-person bounding box and the single pose label in the original data label is changed to the core idea that a single-person bounding box corresponds to two pose labels: for a single-person bounding box bbox i , traverse all human body bounding boxes bbox i except bbox j in the image and calculate the IOU of the two, find the bounding box corresponding to the maximum IOU and the IOU of the two is greater than a preset threshold, then all human body bounding boxes bbox i except bbox j are the maximum covering human body bounding boxes bbox i corresponding to a human body bounding box bbox i_maA_iou ; the human body corresponding to the maximum covering human body bounding box bbox i_maA_iou is regarded as the "crowded person" of the human body corresponding to the human body bounding box bbox i , and the two together constitute a crowded and occluded scene; therefore, the key points of the overlapping part of the maximum covering human body bounding box bbox i_maA_iou and the human body bounding box bbox i need to be extracted as the second pose label label i corresponding to the human body bounding box bbox another . Finally, the original label label corresponding to the human body bounding box bbox i and label another are used together as the human body bounding box bbox ithe pose label of the current person; repeat the above process to traverse each pose label in each image in the dataset and find its corresponding "most crowded" pose label to jointly constitute a supervised label of a crowded-occlusion-aware multi-person pose estimation network model CONet; in addition, if the current person body bounding box bbox i does not exist, then in order to be compatible with the entire training process and the output form of the model, the current label another is taken as the maximum coverage person body bounding box bbox i_maA_iou , that is, it is OK; in order to save computing resources, the above label only needs to be preprocessed once and persisted.

[0125] Performance evaluation index description:

[0126] In this embodiment, the key point similarity (Object Keypoint Similarity, OKS) is taken as the performance evaluation index; for two key point coordinates (x1, y1) and (x2, y2), the key point similarity of the two is calculated as:

[0127]

[0128] wherein p is the current index number of the person; i is the key point index number of the current person; d pi is the Euclidean distance between the two key point coordinates; S p is the scale factor of the person; σ i is the key point normalization factor, which is calculated at the time of creation for a specific dataset; v pi is the visibility of the current key point, which is 0 when not marked, 1 when occluded, and 2 when not occluded, so that it can be seen that the predicted key point that is not marked will not be calculated OKS; for each key point, the similarity result is a value between 0 and 1.

[0129] With the help of the key point similarity OKS, the final performance evaluation index can be calculated: (1) the average detection accuracy (Average Precision, AP) is the average of multiple AP values calculated at different OKS thresholds (.50:.05:.95). (2) the average recall (Average Recall, AR) is the same.

[0130] For the CrowdPose dataset, it defines three different difficulty levels of pose types according to the CrowdIndex: easy (CrowdIndex between 0-0.1), medium (CrowdIndex between 0.1-0.8) and hard (CrowdIndex between 0.8-1), and the corresponding evaluation indexes are AP Easy , APMedSum , AP Hard , represents the crowded degree of key points in a certain human bounding box. Compared with the evaluation indicators of the COCO dataset, the three indicators proposed by the CrowdPose dataset for the Crowd Index can better reflect the algorithm performance in crowded scenes, and AP Hard is a particularly challenging indicator.

[0131] Quantitative results show that:

[0132] CONet will be trained and tested on the CrowdPose and COCO datasets for the multi-person pose estimation task, and compared with other classic methods; as shown in Table 1, the results of the multi-person pose estimation network model based on crowdedness and occlusion perception and existing different models on the pose estimation task of the CrowdPose dataset are shown in Table 1; the higher the indicators, the better the key point results of the corresponding model. By comparing the AP values of each method in the table, it can be seen that the estimation method of the present embodiment has the most advanced performance; and the existing HigherHRNet and PINet are bottom-up methods, so the performance is lower.

[0133] Specifically, HigherHRNet improves the multi-scale problem caused by small heat maps by using the deconvolution module, but it does not have the ability to handle problems in crowded scenes, and it does not explicitly handle the occlusion problem, so it performs poorly on the challenging dataset CrowdPose; OPEC-Net uses graph convolution to infer the occluded key points, but its performance is dependent on the initial pose estimator to some extent, and the initial pose estimator does not handle crowded scenes, so the overall performance is slightly lower; compared with MIPNet, it can also solve the problem of mismatch between model output and crowded scene labels, but since the multi-person output uses the same backbone network and prediction head parameters, it may cause the model to fail in the occlusion scene due to the coupling problem between the occluder and the occluded. The multi-person pose estimation network model CONet based on crowdedness and occlusion perception in the present embodiment explicitly decouples the crowdedness problem, so it can better handle crowded scenes. In addition, MIPNet only provides the results of the backbone network HRNet-W48, which has better performance than HRNet-W32, so if the same backbone network is used, the performance of the estimation method in the present embodiment will be more obvious.

[0134] Table 1 Results of different models on the CrowdPose test set dataset

[0135]

[0136] Qualitative results show that:

[0137] To visually verify that the double branch of the multi-person pose estimation network model CONet based on crowd-occlusion perception indeed establishes different information spaces for different people, in this embodiment, the feature maps of the last layer in the double branch are visualized, i.e., the occluder feature map F out1 in the image to be estimated and the occluded person feature map F out2 in the image to be estimated are visualized; as shown in the accompanying Figure 3 Figures, the visualization results of the occluder feature map F out1 and the occluded person feature map F out2 in the image to be estimated in the embodiment and the multi-person pose estimation results are shown; wherein, Figure 3 (a) is the original image, Figure 3 (b) is the visualization result of the occluder feature map F out1 in the image to be estimated, Figure 3 (c) is the visualization result of the occluded person feature map F out2 in the image to be estimated, Figure 3 (d) is the multi-person pose estimation result; as can be seen from the accompanying Figure 3 Figures, in the multi-person pose estimation network model CONet based on crowd-occlusion perception, the double branch focuses on the features of different people in a complementary manner, thereby verifying that the double branch can indeed decouple two people and can establish different feature spaces for different people in the accompanying Figure 3 Figures; as can be seen from the sub-figures in the second and third rows in the accompanying Figure 3 Figures, when the occlusion is more serious, the model can also pay attention to the key points of different people, which to some extent shows that the attention mechanism plays a role in learning the features of different people.

[0138] Then, the multi-person pose estimation results of CONet and HRNet on the CrowdPose dataset are obtained for qualitative comparison. For the input image, if the actual number of people in each bounding box exceeds 1, it means that the crowd and occlusion in the image are serious. In this chapter, some scenes with serious crowd and occlusion are randomly selected for testing; as shown in the accompanying Figure 4 Figures, the multi-person key point detection results of HRNet, MIPNet and CONet methods in the crowd pose estimation task are shown in the accompanying Figure 4 Figures; each sub-figure is composed of three figures, from left to right, they are the original image with bounding box, the key point detection result of HRNet, and the key point detection result of CONet method.

[0139] By comparing the key point detection results of the three methods, it can be seen that the CONet method has better performance in the crowd and occlusion scenes. Figure 4From the results of the three detection, it can be seen that: (1) the multi-person pose estimation network model CONet based on crowded-occlusion perception is almost not prone to cause the problem of missed detection. When the bounding box contains multiple persons, and the features of other persons are less or their scales are much smaller than that of the main person, only single-person pose prediction results will be produced due to the HRNet only predicting the key points of a single person, and smaller persons are easily ignored by the model, resulting in the key points of secondary persons in the human body detection box being missed; since the double-branch design of CONet defaults to predicting the key point results of two persons, it can naturally perceive multiple persons in the bounding box and thus will not cause the problem of missed detection. (2) The multi-person pose estimation network model CONet based on crowded-occlusion perception is almost not prone to cause the problem of key point mis-matching. When there are multiple persons in the human body bounding box, and the scale difference between the multiple persons is small, and the bounding box contains multiple different human body parts at the same time, since the HRNet takes the multiple persons in the bounding box as a single person input to predict the key point output, the limb part with more obvious features will be predicted as the key point of the current person, and this approach will inevitably give the human body matching the wrong key point. At this time, part of the points will be considered correct when calculating the evaluation index, but from the whole human body pose, the prediction result is far from reality. (3) The multi-person pose estimation network model CONet based on crowded-occlusion perception has small dependence on the quality difference of the generated human body bounding box and is more robust; since the top-down method first detects the human body and then performs pose estimation by taking the cropped human body image as the input, the performance of the top-down method is very sensitive to the quality difference of the human body bounding box; when the human body bounding box is too large, the HRNet (which belongs to the top-down method) is difficult to correctly process the multiple person key points therein; overall, CONet can perceive the crowded situation in the input and decouple the occlusion situation therein to establish a rich information space to process the multiple person key point problem in a single bounding box.

[0140] The description of the related part of the multi-person pose estimation system and device provided in this embodiment can refer to the detailed description of the corresponding part of the multi-person pose estimation method described in this embodiment, which will not be repeated here.

[0141] The multi-person pose estimation method provided in the present application deeply analyzes the problems existing in crowded scenes, and finds that when multiple persons exist in a human bounding box, the current top-down method will fail because it can only detect single-person pose structures; the present application proposes a double-branch structure to model crowded scenes; among them, attention-based convolution operations are used to extract features to improve the ability of each branch to learn different person features and the reasoning ability in occlusion scenes, and accordingly a multi-person pose estimation network model CONet based on crowded-occlusion perception is designed; the effectiveness experiment is carried out on the dataset CrowdPose which has more crowded scenes, and the experimental results show that the multi-person pose estimation network model CONet based on crowded-occlusion perception achieves excellent detection results; the ablation experiment proves that the processing effect of the convolution operation with the attention mechanism on the pose estimation task is better than that of the non local, and the effectiveness of the occlusion guiding mechanism is verified.

[0142] In the present application, a solution is proposed to detect multi-person poses from crowded human bounding boxes, i.e., bounding boxes containing multiple persons, which matches the actual existing multi-person poses in form with the output; a multi-branch structure is used to detect the key points of multiple persons, and each branch is responsible for the pose of a corresponding person, establishing an information space for each person and giving the network the ability to perceive crowded situations; the complex problem of detecting single-person key points from crowded and occluded scenes is decoupled into a "multi-task" problem of predicting the key points of each occluder and occluded person, so that the complex situation of crowded and occluded scenes can be handled by divide and conquer, which is called occlusion perception; a residual connection is established between the occluder and the occluded person to promote the formation of a useful information interaction mechanism between them; the multi-person pose estimation network model based on crowded and occlusion perception can fully model the head structure of crowded and occluded poses, wherein different information spaces are established for the occluded points and the occluded points to handle the occlusion problem through differential modeling.

[0143] In the present application, the crowded and occluded scenes are improved, and a double-branch network architecture CONet based on crowded and occlusion perception is proposed, which establishes an information space for each of the two persons in the crowded bounding box to perceive the crowdedness; the double-branch can also decouple the two persons in the crowded bounding box, which is conducive to the reasoning of the occluded key points of the target person by each branch, thereby perceiving partial occlusion.

[0144] The above embodiment is only one of the implementation manners of the technical scheme of the present application, and the scope of protection claimed by the present application is not limited to the embodiment, but also includes any changes, substitutions and other implementation manners easily thought of by those skilled in the art within the technical scope disclosed by the present application.

Claims

1. A method of multi-person pose estimation, the method comprising: The method comprises the following steps: performing feature extraction on the to-be-estimated image to obtain an input feature map ; obtaining an input feature map of the image to be estimated , as the input of the pre-constructed occluder branch ORB, obtaining an occluder feature map in the image to be estimated ; and using a preset first pose prediction head to perform key point positioning on the occluder feature map in the image to be estimated , to obtain an occluder key point heat map ; perform feature fusion on the input feature map and the occluder feature map to obtain an input feature map perform feature fusion on the input feature map and the occluder feature map to obtain an input feature map perform feature fusion on the input feature map and the occluder feature map to obtain an input feature map ; obtaining an occluded person feature map in the image to be estimated , as the input of the pre-constructed occluded person branch OEB, obtaining an occluded person feature map in the image to be estimated ; and using a preset second pose prediction head to perform key point positioning on the occluded person feature map in the image to be estimated , obtaining an occluded person key point heat map ; keypoint heat map of the occluded person and the keypoint heat map of the occluded person As output, a multi-person pose estimation result of the image to be estimated is obtained.

2. The method of claim 1, wherein, perform feature extraction on the to-be-estimated image to obtain a feature map The process is specifically as follows: The feature extraction is performed on the to-be-estimated image by using a preselected backbone network to obtain a feature map ; wherein the preselected backbone network is an HRNet feature extractor, a ResNet feature extractor, or a SimpleBaseline feature extractor.

3. The method of claim 1, wherein, The pre-constructed occluder branch ORB comprises a first attention module, a first convolution module, a second attention module and a second convolution module connected in sequence; The first attention module is configured to perform attention mechanism operation on the input feature map to obtain an occluder feature map. The first attention module is configured to perform attention mechanism operation on the input feature map to obtain an occluder feature map. The first attention module is configured to perform attention mechanism operation on the input feature map to obtain an occluder feature map. The first convolution module is configured to perform twice convolution operation on the occluder feature map to obtain an occluder feature map ;​ The second attention module is configured to perform an attention mechanism operation on the occluder feature map to obtain an occluder feature map perform an attention mechanism operation to obtain an occluder feature map ; The second convolution module is configured to perform twice convolution operation on the occluder feature map to obtain an occluder feature map in the image to be estimated .

4. The method of claim 3, wherein, The first attention module and the second attention module are the same in structure and each comprises a channel attention module and a spatial attention module; The channel attention module is used to increase the weight of a preset channel; and the spatial attention module is used to increase the weight of a space needing attention.

5. The method of claim 1, wherein, The key point heat map of the occluder is : 。 6. The method of claim 1, wherein, The pre-constructed occludee branch OEB comprises a third attention module, a third convolution module, a fourth attention module and a fourth convolution module connected in sequence; The third attention module is configured to perform attention mechanism operation on the input feature map to obtain an occluded person feature map. perform attention mechanism operation on the input feature map to obtain an occluded person feature map. perform attention mechanism operation on the input feature map to obtain an occluded person feature map. The third convolution module is configured to perform twice convolution operation on the occluder feature map to obtain an occluded person feature map. The third convolution module is configured to perform twice convolution operation on the occluder feature map to obtain an occluded person feature map. The third convolution module is configured to perform twice convolution operation on the occluder feature map to obtain an occluded person feature map. The fourth attention module is configured to perform an attention mechanism operation on the occluder feature map to obtain an occluded person feature map. The fourth attention module is configured to perform an attention mechanism operation on the occluder feature map to obtain an occluded person feature map. The fourth attention module is configured to perform an attention mechanism operation on the occluder feature map to obtain an occluded person feature map. The fourth convolution module is configured to perform twice convolution operation on the occluder feature map to obtain a feature map of an occluded person in the image to be estimated. The fourth convolution module is configured to perform twice convolution operation on the occluder feature map to obtain a feature map of an occluded person in the image to be estimated. The fourth convolution module is configured to perform twice convolution operation on the occluder feature map to obtain a feature map of an occluded person in the image to be estimated 7. The method of claim 6, wherein, The third attention module and the fourth attention module are the same in structure and each comprises a channel attention module and a spatial attention module; The channel attention module is used to increase the weight of a preset channel; and the spatial attention module is used to increase the weight of a space needing attention.

8. The method of claim 1, wherein, Occluded person's keypoint heat map Is: 。 9. A multi-person pose estimation system, comprising: The method comprises: The feature extraction module is configured to perform feature extraction on the to-be-estimated image to obtain an input feature map ; The occluder branch network module is used to process the input feature map. As input to the pre-constructed occluder branch ORB, occluder feature maps are obtained in the image to be estimated. And using a preset first pose prediction head to generate occluder feature maps in the image to be estimated. By locating key points, a key point heatmap of the occluder can be obtained. ; an occlusion guidance module configured to generate an occlusion feature map based on the input feature map and the occluder feature map perform feature fusion to obtain an input feature map ; The occluded branch network module is used to process the input feature map. As input to the pre-constructed Occluded Part of the Edge (OEB) branch, the occluded feature map in the image to be estimated is obtained. And using a preset second pose prediction head to analyze the occluded feature map in the image to be estimated. Keypoint localization was performed to obtain a keypoint heatmap of the occluded subject. ; an output module configured to output the key point heat map of the occluded person and the key point heat map of the occluded person and the key point heat map of the occluded person as output, a multi-person pose estimation result of the image to be estimated is obtained.

10. A multi-person pose estimation device, comprising: The method comprises: A memory for storing a computer program; A processor for executing the computer program to realize the steps of the multi-person pose estimation method according to any one of claims 1-8.

Images