Multi-modal human body posture estimation method based on style conversion

Abstract

The invention discloses a style conversion-based multi-modal human body posture estimation method, which comprises the following steps of: constructing and training a cross-modal style conversion network, and inputting a real visible light image into the trained cross-modal style conversion network to obtain an infrared style image; constructing a multi-modal human body posture reference according to the obtained infrared style image; a modal adaptive attitude estimation network is constructed, a modal specific batch normalization layer is included, the modal style of output is calculated through a Gram matrix, and a modal adaptive loss function is constructed according to the modal style for training; and inputting visible light and infrared images into the trained modal adaptive attitude estimation network to obtain a human body attitude estimation result. According to the embodiment of the invention, the network is transformed in a cross-modal style, so that the network has the capability of generating an infrared image close to the reality; the multi-modal human body posture estimation data set has high-quality labeling; the multi-modal human body posture estimation model has good performance and robustness in a multi-modal scene.

Description

Technical Field

[0001] The present invention relates to the field of human body posture estimation, and in particular to a multimodal human body posture estimation method based on style conversion. Background Art

[0002] The field of human pose estimation (HPE) has been dominated by visible light images. Early studies usually performed single-person pose estimation by directly regressing keypoint coordinates, but the errors of this regression method limited the accuracy and robustness of the model. In contrast, heatmap-based 2D pose estimation methods obtain more robust results by estimating the per-pixel likelihood of each keypoint position and dominate the field of 2D human pose estimation. In recent years, multi-person human pose estimation has attracted increasing attention and posed more realistic challenges. Existing methods can be mainly divided into two categories: top-down and bottom-up:

[0003] The top-down method first detects individuals and then estimates pose keypoints within the detection box. The bottom-up method detects keypoints of all individuals independently and then combines them into a complete skeleton for each person using an association strategy.

[0004] However, these methods usually require sufficient lighting conditions in practical applications, which is not always available in the real world. In low-light conditions, the performance of human pose estimation from visible images degrades significantly.

[0005] In order to solve the problem of insufficient light, human pose estimation in infrared images has attracted much attention in recent years. Currently, research in this field is still limited, and most of the research is inspired by human pose estimation in visible images.

[0006] In the field of human pose estimation, many good benchmarks have been established for human pose estimation from visible light images. Early benchmarks such as LSP and FLIC were crucial in the single-person HPE stage, but were limited by the size of the data. With the emergence of deep learning-based methods, datasets such as MPII aggregated human images in media videos to form a large-scale human pose estimation dataset containing 40,522 annotated images, with the coordinates of 16 joints annotated. Another popular benchmark is COCO, which contains 250,000 annotated individuals, covers a wide range of daily life scenes, and is evaluated using metrics such as AP, AR, and their variants.

[0007] In contrast, there are fewer benchmarks for human pose estimation in infrared images and they are not easy to obtain. The closest to filling this gap is SLP, a multimodal large-scale lying posture dataset containing RGB and infrared modalities, which is mainly used for human posture monitoring in bed.

[0008] To address the lack of HPE benchmarks for infrared images, this paper proposes a new large-scale multimodal human pose estimation benchmark based on MPII and COCO datasets, which can be applied to various real-world scenarios. This new benchmark will significantly promote the development of infrared and visible light multimodal human pose estimation technology.

[0009] On the other hand, the existing style transfer networks for visible light to infrared images are mostly used in the field of re-identification, which pay more attention to the structural clarity of the conversion results and ignore the quality of the conversion results, making the converted infrared images far from the real infrared images.

[0010] In summary, the existing modal style transfer network cannot generate infrared images that are close to reality; the existing multimodal human pose estimation datasets are very scarce, especially in the infrared modality, there is no large-scale and public infrared human pose dataset; the existing human pose estimation methods cannot maintain robustness on multimodal datasets. Summary of the invention

[0011] The present invention aims to address the deficiencies of the prior art and propose a multimodal human posture estimation method based on style transfer.

[0012] The objective of the present invention is to achieve the following technical solution: a multimodal human posture estimation method based on style conversion, the method comprising the following steps:

[0013] S1. Build and train a cross-modal style transfer network, which includes a segmentation network for segmenting instances, a pre-trained cross-modal style transfer module, and an attention map generator;

[0014] S2. Input the real visible light image into the trained cross-modal style transfer network to obtain the infrared style image;

[0015] S3, constructing a multimodal human posture benchmark based on the obtained infrared style image;

[0016] S4. construct a modality-adaptive posture estimation network and use the obtained multimodal human posture benchmark for training, wherein the modality-adaptive posture estimation network includes a modality-specific batch normalization layer, calculates the modality style of the output through the Gram matrix, and constructs a modality adaptation loss function according to the modality style to train the modality-adaptive posture estimation network;

[0017] S5. Input the visible light and infrared images into the trained modality adaptive posture estimation network to obtain the human posture estimation result.

[0018] Furthermore, the cross-modal style transfer network specifically includes: inputting the visible light image into the segmentation network to perform instance segmentation according to the thermal energy level, andi Input to the pre-trained cross-modal style transfer module to convert it into the corresponding infrared style image C i , the attention map generator generates the attention map A of the infrared style image i For fusion weights, infrared style image I irs Obtained by the following formula:

[0019]

[0020] in, represents the Hadamard product, and K is the number of thermal energy levels.

[0021] Furthermore, the training of the cross-modal style transfer network includes: collecting a number of images in a visible light image dataset as visible light domain training images; collecting a number of images in an infrared image dataset as infrared domain training images, preprocessing the collected images and using style loss as training loss to train the network.

[0022] Furthermore, the pre-trained cross-modal style transfer module includes downsampling, 9 ResNet blocks and upsampling, and is pre-trained on RoadScene data.

[0023] Furthermore, the multimodal human posture benchmark includes an original data set and its corresponding infrared style data set.

[0024] Furthermore, the input of the modality adaptive posture estimation network is a visible light image I vis Or infrared style image I irs , the output is the modal feature F mod The specific structure consists of 4 bottlenecks, and all BN layers in each bottleneck are replaced by modality-specific batch normalization layers MSBN; for the output features of the last bottleneck, the gram matrix is ​​calculated to represent the modal style of the features, and the modal adaptation loss function is used to calculate the modal styles of different modalities.

[0025] Furthermore, the modal style is calculated by using the Gram matrix as follows:

[0026] Let G mod ∈R C×C Indicates F mod The feature correlation between the C channel feature maps in , where mod represents the mode, is calculated as follows:

[0027]

[0028] in and They are F at position kmod The activations of the i-th and j-th channels of , mod∈{vis,irs}.

[0029] Furthermore, the modality adaptation loss function is specifically:

[0030]

[0031] Where C and M are the number of channels of the feature map F and the spatial size of the feature.

[0032] On the other hand, the specification also provides a multimodal human posture estimation device based on style transfer, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the multimodal human posture estimation method based on style transfer.

[0033] On the other hand, the specification also provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the multimodal human posture estimation method based on style conversion is implemented.

[0034] The beneficial effects of the present invention are as follows: an example cross-modal style transfer network is proposed, which has the ability to generate infrared images close to the real ones; a large-scale multimodal human pose estimation dataset is proposed with high-quality annotations; a multimodal human pose estimation model is proposed, which has good performance and robustness in multimodal scenarios.

[0035] This paper compares different cross-modal style transfer methods for converting visible light images into infrared style images. When using other methods, the model retains structural information, but cannot effectively perform modal transfer and cannot highlight the brightness differences that infrared images usually show due to inconsistent heat distribution. In contrast, the model of this invention achieves successful modal transfer while retaining a certain degree of structural information.

[0036] The multimodal dataset proposed in the present invention can greatly improve the effect of existing models in multimodal scenarios; when combined with the proposed modal adaptation posture estimation method, the effect is even more significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 An ICSTN structure diagram provided for an embodiment of the present invention;

[0038] Figure 2 A diagram illustrating the CSTM architecture provided in an embodiment of the present invention;

[0039] Figure 3 A partial display diagram of COCO-MM provided in an embodiment of the present invention;

[0040] Figure 4 A diagram illustrating the MAPE architecture provided in an embodiment of the present invention;

[0041] Figure 5 A schematic diagram of a multimodal human posture estimation device based on style conversion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0043] like Figure 1 As shown, the present invention provides a multimodal human posture estimation method based on style conversion, which specifically includes

[0044] S1. Build and train a cross-modal style transfer network, which includes a segmentation network for segmenting instances, a pre-trained cross-modal style transfer module, and an attention map generator; Specifically: The primary goal of the present invention is to establish a large-scale multimodal human posture benchmark. In order to make full use of the existing visual image data source, the present invention proposes a method for generating realistic infrared human posture images from a visual image dataset. Compared with visible light images, infrared images pay more attention to instance-specific information. In human posture estimation, individuals perform significantly differently in infrared intensity than other elements, resulting in fluctuations in brightness levels of different instances in infrared images. However, existing pseudo-infrared generation methods, such as those used for image registration, mainly emphasize structural features and ignore instance-specific changes, resulting in a lack of realism in the generated images.

[0045] Based on the above observations, the present invention proposes an Instance Cross-modal Style Transfer Network (ICSTN). The network consists of two stages for converting the visible light image I vis Convert to infrared style image I irs (like Figure 1 In the first stage, a segmentation network is used Segment the instances in the image according to the thermal energy distribution in the real world. In the implementation of this paper, the instances are divided into three categories by default: high thermal energy instances (e.g., humans, animals), medium thermal energy instances (e.g., cars, motorcycles), and low thermal energy instances (e.g., plants, bicycles). Therefore, the segmentation result I i Related to the thermal energy level. Then, the pre-trained Cross-modal Style Transfer Module (CSTM) is used The visible light image I i Directly converted to the corresponding infrared style image C iIn the second stage, an Attention Map Generator (AMG) based on the UNet architecture was proposed. To generate C i Attention map A i , and use it as the fusion weight. In this process, the style loss is used as the training loss L gen , forcing AMG to generate attention maps to fuse more realistic infrared images. Finally, the infrared style image I irs It can be obtained by the following formula:

[0046]

[0047] Among them, C 0 Represents the visible light input I vis Infrared style image obtained by direct conversion, A 0 Represents and its corresponding attention map; C i Represents infrared style images converted from different heat instances. represents masking these infrared style images, and A i Represents the attention map corresponding to different heat instances; represents the Hadamard product; K is the number of thermal energy levels, which is 3 in this paper; I irs Represents the fused infrared style image.

[0048] The architecture of CSTM is as follows Figure 2 As shown. Specifically, CSTM consists of 3 downsampling convolutional layers, 9 ResNet modules, and 3 upsampling convolutional layers. Infrared information can be introduced. The input is a visible light image I, and the output is an image C with infrared information. CSTM is trained on the RoadScene dataset. During the training of ICSTN on MSRS, the parameters of CSTM are frozen and the parameters of AMG are updated. This ensures that while maintaining the style transfer ability of CSTM, AMG can generate attention maps, thereby producing infrared images that are closer to reality. AMG shares the same architecture as CSTM, but includes 6 ResNet blocks.

[0049] S2. Input the real visible light image into the trained cross-modal style transfer network to obtain the infrared style image;

[0050] S3, constructing a multimodal human posture benchmark based on the obtained infrared style image;

[0051] Through the proposed example cross-modal style transfer network, the present invention is now able to construct large-scale benchmark datasets, named COCO-MM and MPII-MM respectively. Based on the COCO and MPII datasets, the COCO-MM and MPII-MM datasets are constructed respectively, each dataset includes the original dataset and its corresponding infrared style dataset (COCO-IRS and MPII-IRS), so that the scale of each dataset is doubled relative to the original dataset. These datasets provide rich multimodal data, ensure the diversity and richness of the dataset content, and provide sufficient and diverse data support for data-scarce fields. This new benchmark will significantly promote the development of infrared and visible light multimodal human pose estimation technology. Some of the presentations on COCO-MM are as follows: Figure 3 shown.

[0052] S4. Construct a modality-adaptive posture estimation network and use the obtained multimodal human posture benchmark for training, wherein the modality-adaptive posture estimation network includes a modality-specific batch normalization layer, calculates the modality style of the output through the Gram matrix, and constructs a modality adaptation loss function according to the modality style to train the modality-adaptive posture estimation network; specifically:

[0053] Due to the significant differences between visible light and infrared modalities, existing human pose estimation methods are difficult to achieve satisfactory performance in both modalities. In order to address this limitation and achieve accurate human pose estimation for both visible light and infrared modalities, this paper introduces the Modality-Adaptive Pose Estimation (MAPE) method. The structure of MAPE is summarized as follows: Figure 4 As shown in Figure 1, it is mainly composed of Modality-Specific Batch Normalization (MSBN) and Modality-Adaptive Loss (MAL). Specifically, MAPE consists of 4 bottlenecks, and all BN layers in each bottleneck are replaced by MSBN. For the output features of the last bottleneck, the gram matrix is ​​calculated to represent the modal style of the features, and MAL is used to calculate the modal style of different modalities. The input of MAPE is the visible light image I vis Or infrared style image I irs , the output is the visible light feature F vis or F irs (summarized as modal feature F mod). The features processed by MAPE are input into the pose estimator (PoseEstimation) for final pose estimation. By using MSBN, the modality-specific bias can be mitigated during training, allowing the network to capture modality-invariant features more effectively. In addition, a new loss function MAL is introduced to enhance the adaptability of the network to modality changes. Through the integration of MSBN and MAL, MAPE can be easily applied to existing models and achieve robust HPE in multimodal scenarios.

[0054] In each training cycle, the visible light image and its corresponding infrared style image are used together as input. By utilizing different batch normalization in the modality-specific batch normalization, the modality-specific biases are effectively eliminated according to the respective modality. This approach helps the network to learn modality-invariantly, thereby enhancing its adaptability under different modalities and ensuring robust performance in multimodal scenarios.

[0055]

[0056] in μ mod and Represents x mod The mean and variance of the activation values ​​∈ is a constant to prevent division by 0. (γ mod , β mod ) is the affine transformation parameter corresponding to a specific modality in batch normalization. All variables with mod as the subscript can be selected from mod∈(vis, ir), representing batch normalization using different modalities, infrared modality and visible light modality can be selected.

[0057] To achieve efficient information exchange between visible and infrared light modalities, a new loss function, named modality adaptation loss, is introduced. The modality adaptation loss function acts on both visible and infrared branches, promoting the alignment of modality-specific features generated by modality-specific batch normalization. By minimizing feature differences at the input of subsequent pose estimation algorithms, the modality adaptation loss function forces the model to adapt to modality changes, thereby learning modality-invariant representations, enhancing the ability to generalize across different modalities, and ensuring robust performance in multimodal scenarios.

[0058] The present invention uses the Gram matrix to obtain the modality-specific features F after modality-specific batch normalization. mod Compute the modal style. Let G mod ∈R C×C Indicates F mod The feature correlation between the C channel feature maps in , where mod represents the mode. The calculation formula is as follows:

[0059]

[0060] in and They are F at position k mod The activation of the i-th channel and the j-th channel of , in this paper, mod∈{vis,irs}. The modality adaptation loss function can be calculated as follows:

[0061]

[0062] Where C and M are the number of channels of the feature map F and the spatial size of the feature. By minimizing the above loss function, the gap between the modes can be narrowed, helping the model to achieve modal adaptation.

[0063] S5. Input the visible light and infrared images into the trained modality adaptive posture estimation network, and obtain the human posture estimation result through the posture estimator.

[0064] Corresponding to the aforementioned embodiment of a multimodal human posture estimation method based on style conversion, the present invention also provides an embodiment of a multimodal human posture estimation device based on style conversion.

[0065] See also Figure 5 A multimodal human posture estimation device based on style conversion provided by an embodiment of the present invention includes a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a multimodal human posture estimation method based on style conversion in the above embodiment.

[0066] An embodiment of a multimodal human posture estimation device based on style transfer provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 5 As shown, it is a hardware structure diagram of a multi-modal human posture estimation device based on style conversion provided by the present invention, in which any device with data processing capability is located, except Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0067] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0068] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0069] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a multimodal human body posture estimation method based on style conversion in the above embodiment is implemented.

[0070] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0071] The present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the multimodal human posture estimation method based on style conversion is implemented.

[0072] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0073] It should be understood that the above general description and the detailed description below are only exemplary and explanatory and cannot limit the present application. The present application is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the attached claims.

Claims

1. A multimodal human pose estimation method based on style transfer, It is characterized in that The method comprises the following steps: S1. Build and train a cross-modal style transfer network, which includes a segmentation network for segmenting instances, a pre-trained cross-modal style transfer module, and an attention map generator; S2. Input the real visible light image into the trained cross-modal style transfer network to obtain the infrared style image; S3, constructing a multimodal human posture benchmark based on the obtained infrared style image; S4. construct a modality-adaptive posture estimation network, and use the obtained multimodal human posture benchmark for training, wherein the modality-adaptive posture estimation network includes a modality-specific batch normalization layer, calculates the modality style of the output through the Gram matrix, obtains the human posture estimation through the posture estimator, and constructs a modality adaptation loss function according to the modality style to train the modality-adaptive posture estimation network; S5. Input the visible light and infrared images into the trained modality adaptive posture estimation network, and obtain the human posture estimation result through the posture estimator.

2. The method for multimodal human posture estimation based on style transfer according to claim 1, It is characterized in that The cross-modal style transfer network specifically includes: inputting the visible light image into the segmentation network to perform instance segmentation according to the thermal energy level, and i Input to the pre-trained cross-modal style transfer module to convert it into the corresponding infrared style image C i , the attention map generator generates the attention map A of the infrared style image i For fusion weights, infrared style image I irs Obtained by the following formula: in, represents the Hadamard product, and K is the number of thermal energy levels.

3. The method for multimodal human posture estimation based on style transfer according to claim 2, It is characterized in that The training of the cross-modal style transfer network includes: collecting a number of images in a visible light image dataset as visible light domain training images; collecting a number of images in an infrared image dataset as infrared domain training images, preprocessing the collected images and using style loss as training loss to train the network.

4. The method for multimodal human posture estimation based on style transfer according to claim 1, It is characterized in that The pre-trained cross-modal style transfer module includes downsampling, 9 ResNet blocks and upsampling, and is pre-trained on RoadScene data.

5. The multimodal human posture estimation method based on style transfer according to claim 1, It is characterized in that The multimodal human posture benchmark includes an original data set and its corresponding infrared style data set.

6. The multimodal human posture estimation method based on style transfer according to claim 1, It is characterized in that The input of the modality adaptive pose estimation network is a visible light image I vis Or infrared style image I irs , the output is the modal feature F mod The specific structure consists of 4 bottlenecks, and all BN layers in each bottleneck are replaced by modality-specific batch normalization layers MSBN; for the output features of the last bottleneck, the gram matrix is ​​calculated to represent the modal style of the features, and the modal adaptation loss function is used to calculate the modal styles of different modalities.

7. The multimodal human posture estimation method based on style transfer according to claim 1, It is characterized in that The modal style is calculated by using the Gram matrix as follows: Let G mod ∈R C×V Indicates F mod The feature correlation between the C channel feature maps in , where mod represents the mode, is calculated as follows: in and They are F at position k mod The activations of the i-th and j-th channels of , mod∈{vis,irs}.

8. The multimodal human posture estimation method based on style transfer according to claim 7, It is characterized in that The modality adaptation loss function is specifically: Where C and M are the number of channels of the feature map F and the spatial size of the feature.

9. A multimodal human posture estimation device based on style conversion, comprising a memory and one or more processors, wherein the memory stores executable code, It is characterized in that When the processor executes the executable code, a multimodal human posture estimation method based on style transfer according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a program stored thereon, It is characterized in that When the program is executed by a processor, a multimodal human posture estimation method based on style transfer as described in any one of claims 1 to 8 is implemented.

Images