A method, system and medium for multi-person human pose estimation in crowded scenes

Abstract

The application discloses a crowded scene multi-person human posture estimation method, system and medium, and the method comprises the following steps: acquiring a crowded scene picture to be estimated, inputting the crowded scene picture to be estimated into a pre-trained deep learning network model to obtain preliminary features, and then inputting the preliminary features into a preset residual pyramid module to obtain fusion features; inputting the fusion features into a gated self-attention module to obtain global features, and then inputting the global features into an individual information feature extraction module to obtain individual information features corresponding to each individual; inputting the global features and the individual information features into a global spatial decoupling module to obtain deep individual features corresponding to each individual; and inputting the deep individual features into a pre-trained human posture estimation model to obtain human posture estimation results. The application can improve the accuracy and operation speed of the system in crowded scene multi-person human posture estimation, and can be widely applied to the field of human posture estimation.

Description

Technical Field

[0001] This application relates to the field of human pose estimation technology, and in particular to a method, system and medium for estimating the pose of multiple people in crowded scenes. Background Technology

[0002] Existing multi-person human pose estimation algorithms mainly fall into two categories: top-down and bottom-up. The former uses object detection and bounding box algorithms to first obtain multi-person human prediction boxes in crowded scenes, and then performs keypoint detection on each prediction box. The latter uses methods such as Gaussian filtering to obtain keypoint heatmaps, obtains all human keypoint detection results, and then reclassifies and combines these keypoints to obtain a complete human pose estimation result. Regardless of the method, human pose estimation in crowded scenes faces the problem of severe mutual occlusion between the detected targets, making it difficult to match adjacent keypoints to specific human instances. Some single-stage multi-person human pose estimation models directly regress from the overall feature vector to obtain the final result. The feature extraction relied upon by these algorithms is also easily affected by the mutual occlusion and intersection of human targets. Summary of the Invention

[0003] To address the aforementioned technical problems, the present invention aims to provide a method, system, and medium for estimating the pose of multiple human beings in crowded scenarios with high computation speed and high accuracy.

[0004] To achieve the above objectives, one aspect of this application proposes a method for estimating the pose of multiple people in crowded scenes, comprising the following steps:

[0005] Obtain a crowded scene image to be estimated, input the crowded scene image to be estimated into a pre-trained deep learning network model to obtain preliminary features, and then input the preliminary features into a preset residual pyramid module to obtain fused features;

[0006] The fused features are input into the gated self-attention module to obtain global features, and then the global features are input into the individual information feature extraction module to obtain the individual information features corresponding to each individual.

[0007] The global features and the individual information features are input into the global space decoupling module to obtain the deep individual features corresponding to each individual.

[0008] The deep individual features are input into a pre-trained human pose estimation model to obtain the human pose estimation result;

[0009] The human pose estimation results include key point heatmaps and key point detection results for each individual.

[0010] In some embodiments, the step of inputting the preliminary features into a preset residual pyramid module to obtain fused features specifically includes:

[0011] The preliminary features are decomposed using the residual pyramid module to obtain the first input feature and the second input feature;

[0012] The first input feature is resized and extracted to obtain the first output feature;

[0013] The second input feature is decomposed to obtain the third and fourth input features;

[0014] The third and fourth input features are downsampled, bilinearly interpolated, and fused to obtain the second output feature.

[0015] The first output feature and the second output feature are fused together to obtain the fused feature.

[0016] In some embodiments, the global feature is determined by the following formula:

[0017]

[0018]

[0019]

[0020]

[0021] O=(U⊙AV)W o

[0022] Where X represents the fusion feature, W u W v W z and W o Represents the feature dimension transformation matrix. as well as Let X represent the SiLU activation function, U, V, and Z be the intermediate output variables of X, Q and K represent the basic affine transformations, ⊙ represent the Hadamard product, A represent the self-attention matrix, n represent the length of the fused feature, s be the hyperparameter (taken as s = 128), and O represent the global feature.

[0023] In some embodiments, the individual information features are determined by the following formula:

[0024] C=HM center (O)

[0025]

[0026] Ii =(xi,yi)

[0027] f i =F(I i )

[0028] Where O represents the global feature, HM center This indicates the preset heatmap generation module, and C represents the generated heatmap. j This represents the heatmap corresponding to the j-th key point, where m represents the total number of individuals. This represents the center coordinate information of the i-th individual at the j-th key point, and rank represents the maximum value search operation. i Let f represent the center coordinates (xi, yi) of the i-th individual, and let F represent the convolution operation. i This represents the individual information feature corresponding to the i-th individual.

[0029] In some embodiments, the deep individual characteristics are determined by the following formula:

[0030]

[0031] F s i =M i O

[0032]

[0033]

[0034] Where O represents the global feature, f i This represents the individual information feature corresponding to the i-th individual. M represents the channel attention weight feature of the i-th individual. i This represents the spatial attention coefficient of the i-th individual. P represents the spatial location information feature of the i-th individual. i The i-th individual's location coordinates are represented by [], where [] represents the concatenation operation, conv represents the convolution operation, sigmoid represents the non-linear activation function, and F... s i Represents the spatial attention weight feature of the i-th individual, and ReLU represents the nonlinear activation function. The deep individual characteristics represent the i-th individual.

[0035] In some embodiments, the multi-person human pose estimation method further includes a step of pre-training the human pose estimation model, which specifically includes:

[0036] Obtain the Crowdpose dataset;

[0037] The Crowdpose dataset is divided into a training set, a test set, and a validation set. The training set is then preprocessed to obtain dataset label groups.

[0038] The deep learning network is trained based on the dataset label group, the validation set, and the pre-acquired loss function values, and the trained deep learning network is evaluated based on the test set to obtain the human pose estimation model.

[0039] In some embodiments, the training set includes multiple images containing human poses, and the step of preprocessing the training set to obtain a dataset label group specifically includes:

[0040] The training set is subjected to Gaussian filtering to obtain a global heatmap of key points corresponding to the training set.

[0041] Boundary boxes are labeled on each of the images in the training set to obtain the individual center point information corresponding to each human body in each image;

[0042] Gaussian filtering is applied to each of the images in the training set to obtain individual heatmaps of key points corresponding to each human body.

[0043] The dataset label group is obtained by integrating the global heatmap of the key points, the individual center point information, and the individual heatmap of the key points.

[0044] To achieve the above objectives, another aspect of this application proposes a multi-person human pose estimation system for crowded scenarios, comprising:

[0045] The fusion feature acquisition module is used to acquire the crowded scene image to be estimated, input the crowded scene image to be estimated into a pre-trained deep learning network model to obtain preliminary features, and then input the preliminary features into a preset residual pyramid module to obtain fusion features;

[0046] The individual information feature acquisition module is used to input the fused features into the gated self-attention module to obtain global features, and then input the global features into the individual information feature extraction module to obtain the individual information features corresponding to each individual.

[0047] The deep individual feature acquisition module is used to input the global features and the individual information features into the global space decoupling module to obtain the deep individual features corresponding to each individual.

[0048] The human pose estimation module is used to input the deep individual features into a pre-trained human pose estimation model to obtain human pose estimation results.

[0049] The human pose estimation results include key point heatmaps and key point detection results for each individual.

[0050] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the program is executed by the processor, it implements the multi-person human pose estimation method in crowded scenarios as described above.

[0051] To achieve the above objectives, another aspect of the embodiments of this application proposes a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs that can be executed by one or more processors to implement the multi-person human pose estimation method in crowded scenarios as described above.

[0052] The beneficial effects of this invention are as follows: The method, system, and medium for estimating human pose in crowded scenes of this invention obtain preliminary features of the crowded scene image to be estimated through a deep learning network model, obtain fused features through multi-scale fusion using a residual pyramid module, obtain global features through internal correlation capture using a gated self-attention module, obtain deep individual features for estimating human pose through an individual information feature extraction module and a global space decoupling module, and finally input the deep individual features into a pre-trained human pose estimation model to obtain human pose estimation results containing keypoint heatmaps and keypoint detection results for each individual. Through the combined action of the above modules and models, this invention can effectively decouple multiple images in crowded scenes into feature representations of different individual bodies, reducing interference from occlusion and overlap between different bodies on human keypoint detection, more accurately estimating the pose of each body, and improving the accuracy and computational speed of the system for estimating human pose in crowded scenes. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments of the present invention are described below. It should be understood that the drawings described below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart illustrating the steps of a method for estimating the pose of multiple people in a crowded scene, as provided in an embodiment of the present invention;

[0055] Figure 2 This is a network structure diagram of the residual pyramid module provided in an embodiment of the present invention;

[0056] Figure 3 This is a network structure diagram of the gated self-attention module provided in an embodiment of the present invention;

[0057] Figure 4 This is a network structure diagram of the global spatial decoupling module provided in an embodiment of the present invention;

[0058] Figure 5 This is a schematic diagram of key point annotation provided for an embodiment of the present invention;

[0059] Figure 6 This is a schematic diagram of a multi-person human pose estimation system in a crowded scene provided by an embodiment of the present invention;

[0060] Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0062] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0063] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0064] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0065] With the rapid development of computer hardware technology, deep learning, and artificial intelligence algorithms, human pose estimation has always been a hot research topic in the field of computer vision. Human pose estimation can locate key points on the human body, construct human pose models, and has been widely applied in fields such as medical rehabilitation assistance, autonomous driving, and motion analysis. In recent years, with the increasingly widespread application of the Internet of Things, the application scenarios of human pose estimation are increasingly approaching crowded scenarios such as traffic and sports, where multiple people are occluding each other. Research on algorithms for multi-person human pose estimation in such crowded scenarios has a significant impact on the application of pose estimation in modern life.

[0066] Existing multi-person human pose estimation algorithms mainly fall into two categories: top-down and bottom-up. The former uses object detection and bounding box algorithms to first obtain multi-person human prediction boxes in crowded scenes, and then performs keypoint detection on each prediction box. The latter uses methods such as Gaussian filtering to obtain keypoint heatmaps, obtains all human keypoint detection results, and then reclassifies and combines these keypoints to obtain a complete human pose estimation result. Regardless of the method, human pose estimation in crowded scenes faces the problem of severe mutual occlusion between the detected targets, making it difficult to match adjacent keypoints to specific human instances. Some single-stage multi-person human pose estimation models directly regress from the overall feature vector to obtain the final result. The feature extraction relied upon by these algorithms is also easily affected by the mutual occlusion and intersection of human targets.

[0067] To address this issue, this invention proposes a method for estimating the pose of multiple people in crowded scenes. The method involves obtaining preliminary features from the crowded scene image using a deep learning network model, then performing multi-scale fusion using a residual pyramid module to obtain fused features, followed by capturing internal correlations using a gated self-attention module to obtain global features. Next, an individual information feature extraction module and a global space decoupling module are used to obtain deep individual features for estimating human pose. Finally, these deep individual features are input into a pre-trained human pose estimation model to obtain a human pose estimation result containing a keypoint heatmap and keypoint detection results for each individual. Through the combined action of these modules and models, this invention effectively decouples multiple images in crowded scenes into feature representations of different individuals, reducing interference from occlusion and overlap between different people on keypoint detection, thus more accurately estimating the pose of each person and improving the accuracy and computational speed of the system for estimating the pose of multiple people in crowded scenes. This method can be applied to scenarios such as video surveillance, sports competitions, human-computer interaction, medical rehabilitation, autonomous driving, robot navigation, and film production, but is not limited to these applications.

[0068] Reference Figure 1 , Figure 1 This is a flowchart illustrating the steps of a method for estimating the pose of multiple people in a crowded scene, as provided in an embodiment of the present invention. The method includes steps S101 to S104:

[0069] S101. Obtain the crowded scene image to be estimated, input the crowded scene image to be estimated into the pre-trained deep learning network model to obtain preliminary features, and then input the preliminary features into the preset residual pyramid module to obtain fused features.

[0070] Specifically, a deep learning network model pre-trained on the ImageNet database is used as the backbone network for feature extraction to obtain preliminary features of the crowded scene image to be estimated.

[0071] ImageNet is a large visualization database used for research on visual object recognition software. It contains over 14 million image URLs, which are manually annotated to indicate objects in the images, providing bounding boxes in at least one million images;

[0072] The deep learning network model adopted is the HRNet-W32 model. The HRNet-W32 model is a deep learning network model that achieves multi-scale feature extraction and fusion by connecting multiple convolutional layers of different resolutions in parallel. It is mainly used for image recognition and computer vision tasks.

[0073] Reference Figure 2 , Figure 2The network structure diagram of the residual pyramid module is shown below. As an optional implementation, the step of inputting preliminary features into the preset residual pyramid module to obtain fused features can be divided into the following steps S1011 to S1015:

[0074] S1011. The preliminary features are decomposed using the residual pyramid module to obtain the first input feature and the second input feature;

[0075] Specifically, the residual pyramid module includes three 3×3 convolutional layers, three 1×1 convolutional layers, two pooling layers with kernel sizes of 3×3 and 5×5 respectively, and two bilinear interpolation upsampling modules. Each convolutional layer is followed by a batch normalization (BN) layer and a ReLU nonlinear activation layer. The stride and padding of each 3×3 convolutional kernel are both set to 1. The stride determines the distance the convolutional kernel slides across the image, while the padding is used to add extra pixels at the edges of the image.

[0076] S1012. The first input feature is transformed and extracted to obtain the first output feature;

[0077] S1013. Decompose the second input feature to obtain the third and fourth input features;

[0078] S1014. Downsample, bilinearly interpolate, and fuse the third and fourth input features to obtain the second output feature;

[0079] S1015. The first output feature and the second output feature are fused to obtain the fused feature.

[0080] Specifically, the residual pyramid module decomposes the initial features into two parts: a first input feature and a second input feature, categorized by channels. The first input feature is first passed through a 1×1 convolutional layer to change the channel size, and then through a 3×3 convolutional layer for feature extraction. Each convolutional layer is followed by batch normalization (BN) and ReLU nonlinear activation to obtain the first output feature. The second input feature is then fed into the feature pyramid module. The feature pyramid module first passes the second input feature through a 1×1 convolutional layer, then splits it into a third input feature and a fourth input feature. These two parts are then passed through pooling layers with kernel sizes of 3×3 and 5×5, respectively, resulting in feature maps with original feature resolutions of 1 / 2 and 1 / 4. These features are further processed by a 3×3 convolutional layer, and then upsampled using a bilinear interpolation upsampling module to restore their resolution to the previous size. Finally, these two feature parts are added together. Each convolutional layer in this part is also followed by batch normalization (BN) and ReLU nonlinear activation to obtain the second output feature. Finally, the first output feature obtained from the first part and the second output feature obtained from the feature pyramid are added together to obtain the final fused feature X. By processing features through the residual pyramid module, feature maps of different scales can be fused to further enrich the semantic and spatial location information of the features.

[0081] S102. Input the fused features into the gated self-attention module to obtain global features, and then input the global features into the individual information feature extraction module to obtain the individual information features corresponding to each individual.

[0082] Reference Figure 3 , Figure 3 The network structure diagram for the gated self-attention module is shown below. Further, as an optional implementation, the global features are determined by the following formula:

[0083]

[0084]

[0085]

[0086]

[0087] O=(U⊙AV)W o

[0088] Where X represents the fusion feature, W u W v W z and W o Represents the feature dimension transformation matrix. as well as Let represent the SiLU activation function, U, V, and Z represent the intermediate output variables of X, Q and K represent the basic affine transformations, similar to multiplying a and adding b in LN (scaling and translating the normalized value), ⊙ represents the Hadamard product, which is the product of corresponding elements in two matrices, A represents the self-attention matrix, n represents the length of the fused features, s is a hyperparameter, taken as s = 128, and O represents the global features.

[0089] Specifically, the fused feature X obtained through the residual pyramid module is input into the gated self-attention module. The gated self-attention module further obtains the correlation information between local spatial features and global features, while accelerating the system's computation speed and reducing the number of parameters. This allows the fused feature X to integrate relevant information from both local and global aspects, and outputs the global feature O.

[0090] As a further optional implementation, individual information features are determined by the following formula:

[0091] C=HM center (O)

[0092]

[0093] I i =(xi,yi)

[0094] f i =F(I i )

[0095] Where O represents global features, HM center This represents the preset heatmap generation module. The heatmap generation module consists of a convolutional layer with a kernel size of 1×1. The output channel is k+1 of the number of keypoints in the dataset. It locates the heat center of each individual entity using the obtained heatmap. C represents the heatmap generated by the heatmap generation module. j This represents the heatmap corresponding to the j-th key point, where m represents the total number of individuals. This represents the center coordinates of the i-th individual at the j-th keypoint, where rank represents the maximum value search operation. i Let f represent the center coordinates (xi, yi) of the i-th individual, and let F represent the convolution operation. i This represents the individual information feature corresponding to the i-th individual.

[0096] S103. Input the global features and individual information features into the global space decoupling module to obtain the deep individual features corresponding to each individual.

[0097] Reference Figure 4 , Figure 4 The network structure diagram for the global spatial decoupling module is further used as an optional implementation method. Deep individual characteristics are determined by the following formula:

[0098]

[0099] F s i =M i O

[0100]

[0101]

[0102] Where O represents the global feature, f i This represents the individual information feature corresponding to the i-th individual. M represents the channel attention weight feature of the i-th individual. i This represents the spatial attention coefficient of the i-th individual. P represents the spatial location information feature of the i-th individual. i This represents the relevant location coordinate information feature of the i-th individual, [] represents the concatenation operation, conv represents the convolution operation, sigmoid represents the non-linear activation function, and F s i Represents the spatial attention weight feature of the i-th individual, and ReLU represents the nonlinear activation function. This represents the deep individual characteristics of the i-th individual.

[0103] Specifically, the global spatial decoupling module (i.e., the Mask-CBAM module) consists of the channel attention module CAM, the spatial attention module SAM, and the aggregation module Fuse.

[0104] The calculation process expression for the Channel Attention Module (CAM) is as follows:

[0105]

[0106] Where O represents the global feature, f i This represents the individual information feature corresponding to the i-th individual. Since the feature representation of different individuals has different weights in different channels, it can be used to represent the weights of different channels. This indicates element-wise multiplication of matrices. This represents the channel attention weight feature of the i-th individual.

[0107] The calculation process expression for the Spatial Attention Module (SAM) is as follows:

[0108] F s i =M i O

[0109]

[0110] in, This represents the feature representation of the i-th individual, which includes spatial location information. It consists of the global feature O and the individual information feature f corresponding to the i-th individual. i P was obtained. i The feature representation of the relevant location coordinate information of the i-th individual is composed of the center coordinate information I of the i-th individual. i The given information is as follows: [] represents a concatenation operation, conv represents a convolution operation (conv uses a convolutional layer with a kernel size of 5×5, padding of 2, and stride of 1), sigmoid is a non-linear activation function, and M... i M represents the spatial attention coefficient of the i-th individual, obtained through this expression. i That is, it enhances the foreground position representation corresponding to the i-th individual, while weakening the semantic information representation of the remaining positions, F s i This represents the spatial attention weight feature of the i-th individual.

[0111] The calculation process expression for the aggregation module Fuse is as follows:

[0112]

[0113] The convolution operation (conv) is performed using a convolutional layer with a kernel size of 1×1, and ReLU is a non-linear activation function layer. This represents the deep individual features of the i-th individual output by this module, which are used to generate the key point heatmap and key point detection results for the i-th individual.

[0114] By combining the individual information feature extraction module and the global space decoupling module, feature representations focusing on different independent individuals can be obtained, which can increase the system's sensitivity to feature information and improve its performance in estimating the poses of multiple people in crowded scenarios.

[0115] S104. Input the deep individual features into the pre-trained human pose estimation model to obtain the human pose estimation result.

[0116] The human pose estimation results include key point heatmaps and key point detection results for each individual.

[0117] As an optional implementation, the multi-person human pose estimation method further includes a step of pre-training a human pose estimation model, which can be specifically divided into the following steps S1041 to S1043:

[0118] S1041. Obtain the Crowdpose dataset;

[0119] Specifically, the Crowdpose dataset proposes a definition of crowding level for scenes with severe crowd occlusion. The dataset includes 20,000 original images containing human poses, such as... Figure 5 The image shows a schematic diagram of keypoint annotation used in the Crowdpose dataset, in which 14 keypoints are annotated for each human body, including the head, left and right shoulders, left and right elbows, left and right wrists, left and right hip joints, left and right knees, left and right ankles, and the center point of the body.

[0120] S1042. Divide the Crowdpose dataset into a training set, a test set, and a validation set. Then, preprocess the training set to obtain the dataset label group.

[0121] S1043. Train the deep learning network based on the dataset label group, validation set, and pre-acquired loss function values, and evaluate the trained deep learning network based on the test set to obtain the human pose estimation model.

[0122] Specifically, for the pre-acquisition of loss function values, in the individual information feature extraction module, the cross-entropy loss function L1 between different individual information features is calculated, and the cross-entropy loss function L2 between different individual information features and the dataset label group is calculated. In the aggregation module Fuse module, the cross-entropy loss function L3 between the key point heatmap of the i-th individual and the key point individual heatmap of the actual individuals in the dataset label group is calculated. Then, the obtained L1, L2 and L3 are weighted and summed to obtain the loss function value. By training the model parameters through the weighting of different loss functions, the optimal model can be preserved, and the performance and generalization ability of the model can be improved.

[0123] The deep learning network is trained using the dataset label group, validation set, and pre-acquired loss function values. The trained deep learning network is then evaluated using the test set. The Adam optimizer is used to update the network parameters, and a multinomial descent learning rate strategy is used to update the learning rate. The training iteration is 200 epochs, and the model is tested on the validation set each time. The model weights with the smallest loss function value on the validation set are retained to obtain the final human pose estimation model.

[0124] As a further optional implementation, the training set includes multiple images containing human poses. The step of preprocessing the training set to obtain the dataset label group can be specifically divided into the following steps S1044 to S1047:

[0125] S1044. Perform Gaussian filtering on the training set to obtain the global heatmap of key points corresponding to the training set.

[0126] S1045. Mark the bounding boxes of each image in the training set to obtain the individual center point information of each human body in each image.

[0127] S1046. Perform Gaussian filtering on each image in the training set to obtain individual heatmaps of key points for each human body.

[0128] S1047. Integrate the global heatmap of key points, the information of individual center points, and the individual heatmap of key points to obtain the dataset label group.

[0129] Specifically, the training set used for training undergoes annotation preprocessing to obtain data annotations containing more information. A global heatmap of keypoints for the entire input image is obtained using Gaussian filtering. The center point information for each individual in the image is obtained using bounding box annotations, and an individual heatmap of keypoints for each individual is obtained using Gaussian filtering. Finally, the global heatmap of keypoints, the center point information, and the individual heatmaps of keypoints are integrated into a new dataset label group. Simultaneously, the input image size is adjusted to 512×512 to improve computational efficiency.

[0130] The above describes the method for estimating the pose of multiple people in crowded scenes according to embodiments of the present invention. It can be understood that, compared with existing human pose estimation techniques, the embodiments of the present invention obtain preliminary features of the crowded scene image to be estimated through a deep learning network model, obtain fused features through multi-scale fusion using a residual pyramid module to increase the expression of semantic and positional information, and then obtain global features through internal correlation capture using a gated self-attention module to increase the correlation between local and global features. Global features can better express the relationship between each individual and the global context in a crowded scene. Next, deep individual features for estimating human pose are obtained through an individual information feature extraction module and a global space decoupling module. Finally, the deep individual features are input into a pre-trained human pose estimation model to obtain a human pose estimation result containing a keypoint heatmap and keypoint detection results for each individual. Through the combined action of the above modules and models, the present invention can effectively decouple multiple images in crowded scenes into feature representations of different individual people, reducing interference from occlusion and overlap between different people on human keypoint detection, more accurately estimating the pose of each person, and improving the accuracy and computational speed of the system for estimating the pose of multiple people in crowded scenes.

[0131] Reference Figure 6 This invention also provides a multi-person human pose estimation system for crowded scenarios, comprising:

[0132] The fusion feature acquisition module is used to acquire the crowded scene image to be estimated, input the crowded scene image to be estimated into the pre-trained deep learning network model to obtain preliminary features, and then input the preliminary features into the preset residual pyramid module to obtain fusion features;

[0133] The individual information feature acquisition module is used to input the fused features into the gated self-attention module to obtain global features, and then input the global features into the individual information feature extraction module to obtain the individual information features corresponding to each individual.

[0134] The deep individual feature acquisition module is used to input global features and individual information features into the global space decoupling module to obtain the deep individual features corresponding to each individual.

[0135] The human pose estimation module is used to input deep individual features into a pre-trained human pose estimation model to obtain human pose estimation results.

[0136] The human pose estimation results include key point heatmaps and key point detection results for each individual.

[0137] The content of the above embodiments of the multi-person human pose estimation method in crowded scenarios is applicable to this embodiment of the multi-person human pose estimation system in crowded scenarios. The specific functions implemented by this embodiment of the multi-person human pose estimation system in crowded scenarios are the same as those of the above embodiments of the multi-person human pose estimation method in crowded scenarios, and the beneficial effects achieved are also the same as those achieved by the above embodiments of the multi-person human pose estimation method in crowded scenarios.

[0138] This invention also provides an electronic device, comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned method for estimating the human pose of multiple people in crowded scenarios. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0139] like Figure 7 The diagram shown is a hardware structure schematic of an electronic device provided in an embodiment of the present invention. (Refer to...) Figure 7 This invention provides an electronic device, comprising:

[0140] The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0141] The memory 1002 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 to execute the multi-person human pose estimation method in crowded scenes according to the embodiments of the present invention.

[0142] Input / output interface 1003 is used to implement information input and output;

[0143] The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0144] Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004);

[0145] The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0146] This invention also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, which can be executed by one or more processors to implement the above-mentioned method for estimating the pose of multiple people in crowded scenarios.

[0147] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0148] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0149] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the aforementioned blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0150] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the aforementioned functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0151] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0152] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0153] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the aforementioned program can be printed, because the aforementioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or, if necessary, processing in other suitable ways, and then stored in computer memory.

[0154] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0155] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0156] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0157] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for estimating the pose of multiple people in crowded scenes, characterized in that, Includes the following steps: Obtain a crowded scene image to be estimated, input the image into a pre-trained deep learning network model to obtain preliminary features, and then input the preliminary features into a preset residual pyramid module to obtain fused features, including: The preliminary features are decomposed using the residual pyramid module to obtain the first input feature and the second input feature; The first input feature is resized and extracted to obtain the first output feature; The second input feature is decomposed to obtain the third and fourth input features; The third and fourth input features are downsampled, bilinearly interpolated, and fused to obtain the second output feature. The first output feature and the second output feature are fused to obtain the fused feature; The fused features are input into the gated self-attention module to obtain global features, and then the global features are input into the individual information feature extraction module to obtain the individual information features corresponding to each individual. The global features and the individual information features are input into the global space decoupling module to obtain the deep individual features corresponding to each individual. The deep individual features are input into a pre-trained human pose estimation model to obtain the human pose estimation result; The human pose estimation results include key point heatmaps and key point detection results for each individual.

2. The method for estimating the pose of multiple people in a crowded scene according to claim 1, characterized in that, The global features are determined by the following formula: in, This indicates the fusion feature. , , as well as Represents the feature dimension transformation matrix. Represents a non-linear activation function. , as well as express Activation function , as well as for intermediate output variables, and Represents the fundamental affine transformation. Represents the Hadamard product. Represents the self-attention matrix. Indicates the length of the fused feature. For hyperparameters, take , This represents the global feature.

3. The method for estimating the pose of multiple people in a crowded scene according to claim 1, characterized in that, The individual information features are determined by the following formula: in, This represents the global feature. This indicates the preset heatmap generation module. This represents the generated heatmap. Indicates the first Heatmaps corresponding to key points This represents the total number of individuals. Indicates the first The individual mentioned in the first The center coordinates of each key point This indicates a maximum value search operation. Indicates the first The center coordinate information of each individual , This represents the convolution operation. Indicates the first The individual information features corresponding to each individual.

4. The method for estimating the pose of multiple people in a crowded scene according to claim 1, characterized in that, The deep individual characteristics are determined by the following formula: in, Represents the global feature, Indicates the first The individual information features corresponding to each individual. This indicates element-wise multiplication of matrices. Indicates the first Channel attention weight features of each individual, Indicates the first The spatial attention coefficient of each individual. Indicates the first Spatial location information features of the individuals. Indicates the first The relevant location coordinate information features of each individual This indicates a splicing operation. This represents the convolution operation. Represents a non-linear activation function. Indicates the first Spatial attention weight features of the individuals, Represents a non-linear activation function. Indicates the first The deep individual characteristics of the individual.

5. The method for estimating the pose of multiple people in a crowded scene according to claim 1, characterized in that, The multi-person human pose estimation method further includes the step of pre-training the human pose estimation model, which specifically includes: Obtain the Crowdpose dataset; The Crowdpose dataset is divided into a training set, a test set, and a validation set. The training set is then preprocessed to obtain dataset label groups. The deep learning network is trained based on the dataset label group, the validation set, and the pre-acquired loss function values, and the trained deep learning network is evaluated based on the test set to obtain the human pose estimation model.

6. The method for estimating the pose of multiple people in a crowded scene according to claim 5, characterized in that, The training set includes multiple images containing human poses. The step of preprocessing the training set to obtain a dataset label group specifically includes: The training set is subjected to Gaussian filtering to obtain a global heatmap of key points corresponding to the training set. Boundary boxes are labeled on each of the images in the training set to obtain the individual center point information corresponding to each human body in each image; Gaussian filtering is applied to each of the images in the training set to obtain individual heatmaps of key points corresponding to each human body. The dataset label group is obtained by integrating the global heatmap of the key points, the individual center point information, and the individual heatmap of the key points.

7. A multi-person human pose estimation system for crowded scenes, characterized in that, include: The fusion feature acquisition module is used to acquire a crowded scene image to be estimated, input the crowded scene image to be estimated into a pre-trained deep learning network model to obtain preliminary features, and then input the preliminary features into a preset residual pyramid module to obtain fusion features, including: The preliminary features are decomposed using the residual pyramid module to obtain the first input feature and the second input feature; The first input feature is resized and extracted to obtain the first output feature; The second input feature is decomposed to obtain the third and fourth input features; The third and fourth input features are downsampled, bilinearly interpolated, and fused to obtain the second output feature. The first output feature and the second output feature are fused to obtain the fused feature; The individual information feature acquisition module is used to input the fused features into the gated self-attention module to obtain global features, and then input the global features into the individual information feature extraction module to obtain the individual information features corresponding to each individual. The deep individual feature acquisition module is used to input the global features and the individual information features into the global space decoupling module to obtain the deep individual features corresponding to each individual. The human pose estimation module is used to input the deep individual features into a pre-trained human pose estimation model to obtain human pose estimation results. The human pose estimation results include key point heatmaps and key point detection results for each individual.

8. An electronic device, characterized in that, The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for establishing communication between the processor and the memory. When the program is executed by the processor, it implements the steps of the method for estimating the human pose of multiple people in a crowded scene as described in any one of claims 1 to 6.

9. A storage medium, said storage medium being a computer-readable storage medium for computer-readable storage, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the method for estimating the human pose of multiple people in a crowded scene as described in any one of claims 1 to 6.

Images