A multi-modal human pose estimation method based on style conversion

Abstract

The application discloses a kind of multi-modal human posture estimation methods based on style conversion, including building cross-modal style conversion network and training, real visible light image is input to the cross-modal style conversion network trained, and infrared style image is obtained;According to the infrared style image obtained, multi-modal human posture benchmark is constructed;Modal adaptive posture estimation network is constructed, including modal specific batch normalization layer, the output is calculated modal style by Gram matrix, and modal adaptive loss function is constructed according to modal style and is trained;Visible light and infrared image are input to the modal adaptive posture estimation network after training, and human posture estimation result is obtained.The example cross-modal style conversion network of the application has the ability to generate close to real infrared image;Multi-modal human posture estimation dataset has high-quality annotation;Multi-modal human posture estimation model has good performance and robustness in multi-modal scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of human pose estimation, and in particular to a multi-modal human pose estimation method based on style conversion. BACKGROUND

[0002] The field of human pose estimation (HPE) has been dominated by visible light images. Early research usually performs single-person pose estimation by directly regressing keypoint coordinates, but the error of this regression method limits the accuracy and robustness of the model. In contrast, 2D pose estimation methods based on heat maps obtain more robust results by estimating the likelihood of each pixel for each keypoint position, and dominate the field of 2D human pose estimation. In recent years, multi-person human pose estimation has attracted more and more attention, posing more realistic challenges. Existing methods are mainly divided into two categories: top-down and bottom-up:

[0003] Top-down methods first detect individuals, and then estimate pose keypoints within the detection box. Bottom-up methods independently detect the keypoints of all individuals, and then use association strategies to combine them into the complete skeleton of each person.

[0004] However, these methods often require sufficient lighting conditions in practical applications, which are not always available in the real world. When there is insufficient light, the performance of visible image human pose estimation will decrease significantly.

[0005] In order to solve the problem of insufficient light, infrared image human pose estimation has attracted attention in recent years. Currently, research in this field is still limited, and most research is inspired by visible image human pose estimation.

[0006] In the field of human pose estimation, visible light image human pose estimation has established many good benchmarks. Early benchmarks such as LSP and FLIC were crucial in the single-person HPE stage, but were limited by the size of the data. With the advent of deep learning-based methods, datasets such as MPII aggregated human images from media videos, forming a large-scale human pose estimation dataset containing 40,522 annotated images, labeling the coordinates of 16 joints. Another popular benchmark is COCO, containing 250,000 labeled individuals, covering a wide range of daily life scenarios, and using indicators such as AP, AR and their variants for evaluation.

[0007] In contrast, there are fewer and less accessible benchmarks for infrared image human pose estimation. The closest to filling this gap is SLP, a multi-modal large-scale lying pose dataset containing RGB and infrared modalities, mainly used for bed human posture monitoring.

[0008] To solve the problem of lack of infrared image HPE benchmark, the application proposes a new large-scale multi-modal human pose estimation benchmark based on MPII and COCO data sets, which can be applied to various real scenarios. This new benchmark will significantly promote the development of infrared and visible light multi-modal human pose estimation technology.

[0009] On the other hand, existing implementations of visible light to infrared image style conversion networks are mostly used in the field of re-identification, focusing more on the structural clarity of the conversion results, while ignoring the quality of the conversion results, making the converted infrared images far from the real infrared images.

[0010] In summary, the existing modal style conversion network cannot generate infrared images close to the real infrared images; the existing multi-modal human pose estimation data set is very scarce, especially in the infrared modality, there is no large-scale and public infrared human pose data set; the existing human pose estimation method cannot maintain the robustness on the multi-modal data set. SUMMARY

[0011] The application aims to solve the problems of the prior art and proposes a multi-modal human pose estimation method based on style conversion.

[0012] The purpose of the application is achieved by the following technical scheme: a multi-modal human pose estimation method based on style conversion, comprising the following steps:

[0013] S1, build a cross-modal style conversion network and train it, the network includes a segmentation network for segmenting instances, a pre-trained cross-modal style conversion module and an attention map generator;

[0014] S2, input the real visible light image into the trained cross-modal style conversion network to obtain an infrared style image;

[0015] S3, construct a multi-modal human pose benchmark according to the obtained infrared style image;

[0016] S4, construct a modal adaptive pose estimation network, and train it using the obtained multi-modal human pose benchmark, wherein the modal adaptive pose estimation network includes a modal specific batch normalization layer, the output is calculated by a Gram matrix to obtain a modal style, and a modal adaptive loss function is constructed according to the modal style to train the modal adaptive pose estimation network;

[0017] S5, input the visible light and infrared images into the trained modal adaptive pose estimation network to obtain the human pose estimation result.

[0018] Further, the cross-modal style conversion network specifically includes: inputting the visible light image into the segmentation network to segment the instances according to the thermal energy level, segmenting the results Ii input to the pre-trained cross-modal style conversion module to be converted into a corresponding infrared style image C i , the attention map generator generates an attention map A of the infrared style image i for fusing weights, the infrared style image I irs is obtained by the following formula:

[0019]

[0020] wherein, represents the Hadamard product, and K is the number of thermal energy levels.

[0021] Further, the training of the cross-modal style conversion network comprises: collecting a plurality of images in a visible light image dataset as visible light domain training images; collecting a plurality of images in an infrared image dataset as infrared domain training images, pre-processing the collected images, and using a style loss as a training loss to train the network.

[0022] Further, the pre-trained cross-modal style conversion module comprises down-sampling, 9 ResNet blocks and up-sampling, and is pre-trained on RoadScene data.

[0023] Further, the multi-modal human pose reference comprises an original dataset and a corresponding infrared style dataset thereof.

[0024] Further, the modality adaptive pose estimation network inputs a visible light image I vis or an infrared style image I irs , and outputs a modality feature F mod , and the specific structure comprises 4 bottlenecks, all BN layers in each bottleneck are replaced with a modality-specific batch normalization layer MSBN; for the output feature of the last bottleneck, a gram matrix is calculated to represent the modality style of the feature, and a modality adaptive loss function is used to calculate the modality style of different modalities.

[0025] Further, the calculation of the modality style through the gram matrix is specifically as follows:

[0026] Let G mod ∈R C×C represent the feature correlation between C channel feature maps in F mod , wherein mod represents the modality, and the calculation formula is as follows:

[0027]

[0028] wherein, and are Fmod activation of the i-th channel and the j-th channel of the modality, mod e {vis,irs}.

[0029] Further, the modality adaptive loss function is specifically:

[0030]

[0031] where C and M are the number of channels and the spatial size of the features of the feature map F.

[0032] In another aspect, the specification also provides a multi-modal human pose estimation device based on style conversion, comprising a memory and one or more processors, the memory stores executable code, and the processor executes the executable code to implement the multi-modal human pose estimation method based on style conversion.

[0033] In another aspect, the specification also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the multi-modal human pose estimation method based on style conversion.

[0034] The beneficial effects of the present application are: an example cross-modal style conversion network is proposed, which has the ability to generate images close to real infrared images; a large-scale multi-modal human pose estimation dataset is proposed, which has high-quality annotations; a multi-modal human pose estimation model is proposed, which has good performance and robustness in multi-modal scenarios.

[0035] The present application compares different cross-modal style conversion methods for converting visible light images into infrared style images. When using other methods, the model retains structural information, but cannot effectively perform modal conversion and cannot highlight the brightness difference that infrared images usually exhibit due to inconsistent heat distribution. In contrast, the model of the present application achieves successful modal conversion while retaining a certain degree of structural information.

[0036] The multi-modal dataset proposed by the present application can greatly improve the effect of existing models in multi-modal scenarios; when combined with the proposed modality adaptive pose estimation method, the effect is improved more obviously. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The ICSTN structure diagram provided for the embodiments of the present application;

[0038] Figure 2 The CSTM architecture diagram provided for the embodiments of the present application;

[0039] Figure 3 The partial display diagram of COCO-MM provided for the embodiments of the present application;

[0040] Figure 4 MAPE architecture diagram provided for an embodiment of the present application;

[0041] Figure 5 Style conversion-based multi-modal human pose estimation device schematic diagram provided for an embodiment of the present application. DETAILED DESCRIPTION

[0042] The specific embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0043] As Figure 1 shown, the present application provides a style conversion-based multi-modal human pose estimation method, which specifically includes

[0044] S1, build a cross-modal style conversion network and train it, which includes a segmentation network for segmenting instances, a pre-trained cross-modal style conversion module and an attention map generator; specifically: the primary goal of the present application is to establish a large-scale multi-modal human pose benchmark. In order to make full use of existing visual image data sources, the present application proposes a method for generating realistic infrared human pose images from visual image data sets. Compared with visible light images, infrared images pay more attention to instance-specific information. In human pose estimation, the performance of individuals on infrared intensity is significantly different from other elements, resulting in fluctuations in the brightness level of different instances in infrared images. However, existing pseudo-infrared generation methods, such as those used for image registration, mainly emphasize structural features and ignore instance-specific changes, resulting in a lack of realism in the generated images.

[0045] According to the above observation, the present application proposes an instance cross-modal style conversion network (Instance Cross-modal Style Transfer Network, ICSTN). The network includes two stages for converting visible light images I vis into infrared style images I irs (as Figure 1 shown). In the first stage, a segmentation network is used to segment instances in the image according to the thermal energy distribution in the real world. In the implementation herein, it is defaulted to be divided into three categories: high thermal energy instances (e.g. humans, animals), medium thermal energy instances (e.g. cars, motorcycles) and low thermal energy instances (e.g. plants, bicycles). Therefore, the segmentation result I i is related to the thermal energy level. Then, a pre-trained cross-modal style conversion module (Cross-modal Style Transfer Module, CSTM) is used to directly convert the visible light image I i into the corresponding infrared style image C iIn the second stage, an Attention Map Generator (AMG) based on the UNet architecture was proposed. Used to generate C i Attention map A i This is used as the weight for fusion. In this process, style loss is used as the training loss L. gen This forces AMG to generate an attention map to fuse a more realistic infrared image. Ultimately, the infrared-style image I... irs It can be obtained from the following formula:

[0046]

[0047] Wherein, C0 represents the input of visible light I vis The infrared style image obtained through direct conversion, A0 represents and its corresponding attention map; C i Infrared style images representing different heat instances converted from different thermal instances. This represents masking these infrared-style images, while A i Attention maps representing different heat instances; Represents the Hadamard product; K is the number of thermal energy levels, which is 3 in this paper; I irs This represents the fused infrared-style image.

[0048] The architecture of CSTM is as follows Figure 2 As shown. Specifically, CSTM consists of 3 downsampling convolutional layers, 9 ResNet modules, and 3 upsampling convolutional layers. Infrared information can be incorporated. The input is a visible light image I, and the output is an image C with infrared information. CSTM is trained on the RoadScene dataset. During the training of ICSTN on MSRS, the parameters of CSTM are frozen, and the parameters of AMG are updated. This ensures that while maintaining the style transfer capability of CSTM, AMG can generate attention maps, thus producing infrared images that are closer to reality. AMG shares the same architecture as CSTM but includes 6 ResNet blocks.

[0049] S2. Input the real visible light image into the trained cross-modal style transfer network to obtain the infrared style image;

[0050] S3. Construct a multimodal human posture reference based on the obtained infrared style images;

[0051] Through the proposed example cross-modal style conversion network, the present application can now construct large-scale benchmark datasets, named COCO-MM and MPII-MM respectively. Based on the COCO and MPII datasets, the COCO-MM and MPII-MM datasets are constructed respectively, each of which includes the original dataset and its corresponding infrared style dataset (COCO-IRS and MPII-IRS), so that the size of each dataset is doubled relative to the original dataset. These datasets provide rich multi-modal data, ensuring the diversity and richness of the dataset content, and providing sufficient and diverse data support for data-scarce fields. This new benchmark will significantly promote the development of infrared and visible light multi-modal human pose estimation technology. Part of the COCO-MM demonstration is shown as Figure 3 .

[0052] S4, constructing a modality adaptive pose estimation network, training the modality adaptive pose estimation network using the obtained multi-modal human pose benchmark, wherein the modality adaptive pose estimation network comprises a modality-specific batch normalization layer, calculating the modality style by Gram matrix for the output, and training the modality adaptive pose estimation network according to the modality style constructed modality adaptive loss function; specifically:

[0053] Due to the significant difference between visible light and infrared modalities, existing human pose estimation methods are difficult to achieve satisfactory performance on both modalities. In order to solve this limitation and achieve accurate human pose estimation for visible light and infrared modalities, the present application introduces a modality adaptive pose estimation method (Modality-Adaptive Pose Estimation, MAPE). The structure of MAPE is shown as Figure 4 , mainly composed of modality-specific batch normalization (Modality-Specific Batch Normalization, MSBN) and modality adaptive loss function (Modality-Adaptive Loss, MAL). Specifically, MAPE is composed of 4 bottlenecks, and all BN layers in each bottleneck are replaced by MSBN. For the output features of the last bottleneck, gram matrix is calculated to represent the modality style of the features, and MAL is used to calculate the modality style of different modalities. The input of MAPE is a visible light image I vis or an infrared style image I irs , and the output is a visible light feature F vis or F irs (summarized as modality feature F mod) is used for the final pose estimation. By using MSBN, the modal-specific bias can be alleviated during training, making the network capture the modal-invariant features more effectively. Moreover, a new loss function MAL is introduced to enhance the adaptability of the network to the modal variations. With the integration of MSBN and MAL, MAPE can be easily applied to existing models and achieve robust HPE in the multi-modal scenarios.

[0054] In each training cycle, the visible light image and its corresponding infrared style image are used as inputs together. By utilizing different batch normalization in the modal-specific batch normalization, the modal-specific bias can be effectively eliminated according to the respective modal. This method helps the network to learn modal-invariant, thereby enhancing its adaptability to different modal and ensuring the robust performance in the multi-modal scenarios.

[0055]

[0056] where μ mod and denote the mean and variance of the activation values of x mod , and ∈ is a constant to prevent division by 0. (γ mod , β mod ) are the affine transformation parameters corresponding to the specific modal in batch normalization. All variables subscripted by mod can be selected from mod ∈ (vis, ir), representing the batch normalization using different modal, and the infrared modal and the visible light modal can be selected.

[0057] To achieve efficient information exchange between the visible light and infrared light modal, a new loss function named modal adaptation loss function is introduced. The modal adaptation loss function acts on both the visible light and infrared light branches, promoting the alignment of the specific modal features generated by the modal-specific batch normalization. By minimizing the feature difference at the input end of the subsequent pose estimation algorithm, the modal adaptation loss function promotes the model to adapt to the modal variations, thereby learning the modal-invariant representation, enhancing the ability to generalize across different modal, and ensuring the robust performance in the multi-modal scenarios.

[0058] The present application utilizes the Gram matrix to calculate the modal style according to the modal-specific batch normalization processed modal-specific features F mod . Let G mod ∈ R C×C denote the feature correlation between C channel feature maps in F mod , where mod represents the modal. The calculation formula is as follows:

[0059]

[0060] in and F at position k mod The activations of the i-th and j-th channels are given, where mod ∈ {vis,irs}. The modal adaptation loss function can be calculated as follows:

[0061]

[0062] Here, C and M represent the number of channels in the feature map F and the spatial size of the features, respectively. By minimizing the above loss function, the gap between modes can be reduced, helping the model achieve modal adaptation.

[0063] S5. Input the visible light and infrared images into the trained modality adaptive pose estimation network, and obtain the human pose estimation result through the pose estimator.

[0064] Corresponding to the aforementioned embodiment of a style-transfer-based multimodal human pose estimation method, the present invention also provides an embodiment of a style-transfer-based multimodal human pose estimation device.

[0065] See Figure 5 The present invention provides a multimodal human pose estimation device based on style transfer, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a multimodal human pose estimation method based on style transfer in the above embodiment.

[0066] An embodiment of a style-transfer-based multimodal human pose estimation device provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data-processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 5 The diagram shown is a hardware structure diagram of any device with data processing capabilities, in which the style transfer-based multimodal human pose estimation device provided by this invention is located. (Except for...) Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0067] The implementation process of the functions and roles of each unit in the above device is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0068] For the device embodiment, since it basically corresponds to the method embodiment, the relevant part can be referred to the part of the method embodiment. The device embodiments described above are only illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present application according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0069] The embodiment of the application further provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the multi-modal human pose estimation method based on style conversion in the above embodiment.

[0070] The computer readable storage medium can be an internal storage unit of any data processing capable device, such as a hard disk or a memory. The computer readable storage medium can also be an external storage device of any data processing capable device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both an internal storage unit and an external storage device of any data processing capable device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing capable device, and can also be used to temporarily store data that has been output or will be output.

[0071] The application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to realize the multi-modal human pose estimation method based on style conversion.

[0072] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The specification and examples given are exemplary only and the true scope and spirit of the application is indicated by the claims. The specification and examples are to be regarded as illustrative only, the true scope and spirit of the application being indicated by the claims.

[0073] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application, as claimed. The application is not restricted to the exact details shown and described herein and that many variations and modifications can be made thereto, without departing from the scope of the present application. The scope of the present application should only be determined with reference to the appended claims.

Claims

1. A multimodal human pose estimation method based on style transfer, characterized in that, The method includes the following steps: S1. Build and train a cross-modal style transfer network, which includes a segmentation network for segmenting instances, a pre-trained cross-modal style transfer module, and an attention map generator; The cross-modal style transfer network specifically includes: inputting a visible light image into a segmentation network to perform instance segmentation based on thermal energy levels, and then processing the segmentation result I... i The input is fed into a pre-trained cross-modal style transfer module and converted into the corresponding infrared style image C. i The attention map generator generates attention maps A for infrared-style images. i Used for weight fusion, infrared style image I irs It is obtained from the following formula: in, This represents the Hadamard product, where K is the number of thermal energy levels. This involves masking these infrared-style images; S2. Input the real visible light image into the trained cross-modal style transfer network to obtain the infrared style image; S3. Construct a multimodal human posture reference based on the obtained infrared style images; S4. Construct a modal adaptive pose estimation network and train it using the obtained multimodal human pose benchmark. The modal adaptive pose estimation network includes a modality-specific batch normalization layer. The output is used to calculate the modal style through the Gram matrix. The human pose is estimated through the pose estimator. The modal adaptive pose estimation network is trained by constructing a modal adaptation loss function based on the modal style. The calculation of modal style using the Gram matrix specifically involves: Let G mod ∈R C×C F represents mod The feature correlations among the C channel feature maps, where mod represents the modality, are calculated using the following formula: in and F at position k mod The activation of the i-th and j-th channels, mod∈{vis,irs}; The modal adaptation loss function is specifically as follows: Where C and M are the number of channels in the feature map F and the spatial size of the feature; S5. Input the visible light and infrared images into the trained modality adaptive pose estimation network, and obtain the human pose estimation result through the pose estimator.

2. The multimodal human pose estimation method based on style transfer according to claim 1, characterized in that, The training of the cross-modal style transfer network includes: collecting several images from the visible light image dataset as visible light domain training images; collecting several images from the infrared image dataset as infrared domain training images; performing preprocessing on the collected images; and using style loss as the training loss to train the network.

3. The multimodal human pose estimation method based on style transfer according to claim 1, characterized in that, The pre-trained cross-modal style transfer module includes downsampling, nine ResNet blocks, and upsampling, and is pre-trained on RoadScene data.

4. The multimodal human pose estimation method based on style transfer according to claim 1, characterized in that, The multimodal human pose benchmark includes the original dataset and its corresponding infrared style dataset.

5. The multimodal human pose estimation method based on style transfer according to claim 1, characterized in that, The modality adaptive pose estimation network is input to a visible light image I. vis Or infrared style image I irs The output is the modal feature F. mod The specific structure consists of 4 bottlenecks. In each bottleneck, all BN layers are replaced with mode-specific batch normalized layers (MSBN). For the output features of the last bottleneck, a gram matrix is ​​calculated to represent the modal style of the features, and the modal style of different modes is calculated using the modal adaptation loss function.

6. A style-transfer-based multimodal human pose estimation device, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a multimodal human pose estimation method based on style transfer as described in any one of claims 1-5.

7. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a multimodal human pose estimation method based on style transfer as described in any one of claims 1-5.

Images