Multi-modal pedestrian detection method

Abstract

The invention discloses a multi-modal pedestrian detection method, and the method comprises the steps: an imaging quality sensing module which is used for sensing the imaging quality difference of different regions of a visible light image and an infrared image, and mining the complementary characteristics of a multi-modal image; the multi-modal feature fusion module is used for guiding feature fusion of the visible light-infrared multi-modal image by using the normalized imaging quality spatial distribution map to obtain multi-modal fusion feature representation; the key area guiding feature enhancement module predicts a key area where a small-scale pedestrian is located and performs feature enhancement on the key area to obtain fusion feature representation with higher discrimination; and sending the enhanced fusion features to a detection head to obtain a pedestrian prediction result. According to the method, more comprehensive visible light-infrared multi-mode feature representation is obtained, so that the accuracy of pedestrian detection is improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and pedestrian detection, and in particular to a multimodal pedestrian detection method. Background Art

[0002] Accurately detecting pedestrian targets in drone aerial photography scenes can effectively improve pedestrian search efficiency in complex scenes such as mountainous areas and is widely used in tasks such as field rescue. However, in actual application scenarios, there is a need for pedestrian detection throughout the day, and factors such as high exposure during the day and low illumination at night inevitably affect the accuracy of pedestrian detection. Considering that visible light imaging has the ability to describe detailed information such as scene color and texture, while infrared imaging has the characteristics of thermal radiation perception, the combined use of visible light images and infrared images can effectively achieve all-day pedestrian detection. Therefore, exploring a combined visible light-infrared multimodal pedestrian detection method to obtain accurate pedestrian detection results by fully integrating multimodal complementary information has important research significance and application value.

[0003] In recent years, researchers have proposed a series of visible-light-infrared multimodal pedestrian detection methods, which effectively fuse multimodal information to achieve high-quality pedestrian detection. Zhang et al. proposed a pedestrian detection method based on overall illumination perception. By sensing image-level illumination changes, they fuse multimodal information under different illumination conditions to improve pedestrian detection performance. Chen et al. proposed an attention alignment network. By focusing the network on pedestrian areas and alleviating the impact of environmental illumination imbalance on multimodal information fusion, they can effectively detect pedestrian targets in a scene.

[0004] Most of the aforementioned methods assess the overall imaging quality differences between visible and infrared images by perceiving the overall illumination conditions of images from different modalities, thereby achieving visible-infrared feature fusion. However, in aerial imagery, due to the wide coverage of the scene and complex backgrounds, image quality imbalances also exist across different regions of visible and infrared images. Furthermore, the relatively small scale of pedestrians in aerial scenes makes small-scale pedestrian features susceptible to interference from complex background features during image feature extraction, which in turn affects the network's ability to effectively detect pedestrians and limits further improvements in pedestrian detection performance. Summary of the Invention

[0005] The present invention provides a multimodal pedestrian detection method. By sensing regional differences in multimodal image quality and enhancing features in key areas where small-scale targets are located, the present invention fully exploits the complementary nature of multimodality and enhances features in key areas. This alleviates the adverse effects of visible-light-infrared imaging quality imbalance and the small size of pedestrians in aerial photography scenes, and obtains a more comprehensive visible-light-infrared multimodal feature representation, thereby improving the accuracy of pedestrian detection. Detailed description is provided below:

[0006] A multimodal pedestrian detection method, comprising:

[0007] The imaging quality perception module is used to perceive the differences in imaging quality between different regions of visible light and infrared images and to exploit the complementary characteristics of multimodal images.

[0008] The multimodal feature fusion module uses the normalized imaging quality spatial distribution map to guide the feature fusion of visible light and infrared multimodal images to obtain a multimodal fusion feature representation;

[0009] The key region guided feature enhancement module predicts the key regions where small-scale pedestrians are located and enhances the features of the key regions to obtain a more discriminative fused feature representation;

[0010] The enhanced fusion features are fed into the detection head to obtain pedestrian prediction results.

[0011] Wherein, the imaging quality perception module is:

[0012] Visible light image I v and infrared image I t As input, each modality image is evenly divided into m×n subregions;

[0013] Using a vision-language basic model, the imaging quality of each sub-region of the multimodal image is evaluated. The evaluation results of each sub-region are integrated to obtain the spatial distribution map of the imaging quality of the visible light image and the infrared image.

[0014] The spatial distribution diagram of imaging quality of each modality was normalized.

[0015] Among them, the spatial distribution diagram of imaging quality is:

[0016]

[0017] Among them, Q v and Q t Represent the spatial distribution diagrams of imaging quality of visible light images and infrared images respectively, and Represent the imaging quality assessment results of the first sub-region of the visible light image and infrared image, respectively. and Represent the imaging quality assessment results of the m×nth sub-region of the visible light image and infrared image respectively, VL() represents the vision-language basic model, and prompt represents the language prompt.

[0018] Among them, the normalized imaging quality spatial distribution map Q' v and Q tUnder the guidance of ', the two modal features are fused to give full play to the complementary advantages of the two modal information. The fusion process is as follows:

[0019]

[0020] Among them, F v and F t Represent the original features of the two modal images respectively, F represents the multimodal fusion feature representation, represents element-wise matrix multiplication, represents element-by-element addition of matrices, cat(·) represents feature concatenation, and conv(·) represents a convolutional layer with a convolution kernel size of 1×1.

[0021] The key area guided feature enhancement module is:

[0022] For the input multimodal fusion feature representation F, first input the key area prediction unit composed of the initialization layer, the spatial transformation encoder and the channel compression layer to obtain the key area prediction response map W where the small-scale pedestrian is located;

[0023] Calculate the absolute scale D of each ground truth box abs , select the real box whose absolute scale is smaller than the threshold ε and represent the key area where it is located as a Gaussian mask map;

[0024] Cross entropy loss is used to constrain the learning of key region prediction units, and the response map is used to enhance the features of small-scale pedestrians.

[0025] The process of selecting a true box whose absolute scale is smaller than the threshold ε and representing the key area where it is located as a Gaussian mask is as follows:

[0026]

[0027] Among them, i and j represent the row and column index of the mask image respectively, and x k and y k They jointly represent the center coordinates of the kth true box, and K represents the number of true boxes whose absolute scale is smaller than the threshold ε.

[0028] The method of enhancing the small-scale pedestrian features using the response map is as follows: scaling W to the same size as the fused feature F by maximum pooling, and enhancing the fused feature F to obtain the enhanced fused feature F'. The feature enhancement process is shown as follows:

[0029]

[0030] Among them, MaxPool(·) represents the maximum pooling operation, represents element-wise matrix multiplication, Represents element-wise addition of matrices.

[0031] The beneficial effects of the technical solution provided by the present invention are:

[0032] 1. This invention obtains accurate pedestrian detection results in aerial photography scenes by effectively fusing visible light and infrared multimodal complementary information and enhancing key area features;

[0033] 2. The present invention designs an imaging quality perception module that effectively exploits the complementary characteristics of multimodal images by perceiving the differences in imaging quality between different regions of visible light and infrared images.

[0034] 3. The present invention designs a key area guided feature enhancement module, which predicts the key areas where small-scale pedestrians are located and effectively enhances the key area features to obtain a more discriminative feature representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Flowchart of a multimodal pedestrian detection method. DETAILED DESCRIPTION

[0036] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0037] 1. Constructing a dual-branch feature extraction network

[0038] In order to obtain the feature representation of visible light-infrared multimodal images, the embodiment of the present invention constructs a dual-branch feature extraction network.

[0039] Specifically, the paired visible light image and infrared image are represented as I v and I t Then, a dual-branch network with the same structure is used to extract the visible light image I v and infrared image I t The feature representation of the process is as follows:

[0040] F v =Backbone v (I v )

[0041] F t =Backbone t (I t )

[0042] Among them, Backbone v () and Backbone t () represent the feature extraction networks of visible light modal branch and infrared modal branch respectively, F v and F tRepresenting visible light image features and infrared image features, respectively. In this embodiment of the present invention, ResNet50 is used as the backbone network, and the backbone network parameters are initialized using a model pre-trained on the ImageNet dataset. The ResNet50 backbone network and the ImageNet dataset are both well known in the art and will not be described in detail here.

[0043] 2. Design of the Image Quality Perception Module

[0044] In order to measure the imaging quality of each area of ​​a multimodal image and thereby guide the subsequent multimodal feature fusion process, an embodiment of the present invention designs an imaging quality perception module. By perceiving the differences in imaging quality between different areas of visible light images and infrared images, it effectively exploits the complementary characteristics of multimodal images.

[0045] Specifically, the imaging quality perception module uses the visible light image I v and infrared image I t As input. First, the same method is used to process the visible light image I v and infrared image I t The average region division is performed separately, and each modality image is divided into m×n sub-regions. The process is expressed as follows:

[0046]

[0047] in, and Represent the first sub-region of the visible light image and infrared image respectively, and so on and In the embodiment of the present invention, m and m are both set to 5.

[0048] Subsequently, to obtain the imaging quality distribution of multimodal images, a vision-language based model was used to evaluate the imaging quality of each subregion of the multimodal image by constructing corresponding language cues. The evaluation results of each subregion were then integrated to obtain the spatial distribution map of the imaging quality of the visible light image and the infrared image. The process is shown as follows:

[0049]

[0050] Among them, Q v and Q t Represent the spatial distribution diagrams of imaging quality of visible light images and infrared images respectively, and Represents the imaging quality evaluation results of the first sub-region of the visible light image and infrared image, and so on and represents the imaging quality assessment results for the m×nth subregion of the visible light image and infrared image, respectively. VL() represents the visual-language foundation model, and prompt represents the language prompt. In this embodiment of the present invention, CLIP is used as the visual-language foundation model, and the language prompt is set to "high-quality, appropriately bright, and clear image."

[0051] The imaging quality spatial distribution diagram Q of visible light image and infrared image is obtained v and Q t Afterwards, the spatial distribution diagram of imaging quality of each modality is normalized. The formula for the normalization operation is as follows:

[0052]

[0053] Among them, Q' v and Q t 'represent the spatial distribution diagrams of imaging quality after normalization of the two modalities.

[0054] 3. Design of multimodal feature fusion module

[0055] In order to effectively fuse the visible light and infrared multimodal complementary information, the embodiment of the present invention designs a multimodal feature fusion module, which uses the normalized imaging quality spatial distribution map to guide the feature fusion of visible light and infrared multimodal images, and then obtains a multimodal fusion feature representation.

[0056] Specifically, in the normalized imaging quality spatial distribution map Q' v and Q t Under the guidance of ', the two modal features are effectively fused to give full play to the complementary advantages of the two modal information. The fusion process is expressed as follows:

[0057]

[0058] Among them, F v and F t Represent the original features of the two modal images respectively, F represents the multimodal fusion feature representation, represents element-wise matrix multiplication, represents element-by-element addition of matrices, cat(·) represents feature concatenation, and conv(·) represents a convolutional layer with a convolution kernel size of 1×1.

[0059] 4. Designing a Key Area-Guided Feature Enhancement Module

[0060] In order to effectively enhance the features of the key areas where small-scale pedestrians are located, an embodiment of the present invention designs a key area guided feature enhancement module, which predicts the key areas where small-scale pedestrians are located and enhances the features of the key areas to obtain a more discriminative fusion feature representation.

[0061] For the input multimodal fusion feature representation F, first input the key area prediction unit consisting of the initialization layer, the spatial transformation encoder and the channel compression layer to obtain the key area prediction response map W where the small-scale pedestrian is located. The formula is expressed as:

[0062] W=sig(ch(sw(init(F))))

[0063] Among them, init(·) represents the initialization layer, which is composed of a convolution layer with a convolution kernel size of 3×3. sw(·) represents the spatial transform encoder, which contains N spatial transform blocks, where is the ratio of the original image size to the feature map size. Each spatial transformation block consists of a 3×3 convolutional layer, a batch normalization layer, a ReLU activation layer, and an upsampling layer, where the upsampling layer is implemented via bilinear interpolation. ch(·) represents a channel compression layer, consisting of a 3×3 convolutional layer with one output channel. sig(·) represents the sigmoid activation function, which is used to map the values ​​of W to the range (0, 1).

[0064] Subsequently, in order to ensure that the response map can effectively reflect the distribution of key areas of small-scale pedestrians, the real box of small-scale pedestrian targets is used to construct constraints. Specifically, the absolute scale D of each real box is first calculated. abs , the process is as follows:

[0065]

[0066] Among them, w and h are the width and height of the pedestrian's real frame, respectively, both in pixels.

[0067] After obtaining the absolute scale of each true box, we use it as a metric to select the true box whose absolute scale is less than the threshold ε and represent the key area where it is located as a Gaussian mask M. The process is as follows:

[0068]

[0069] Among them, i and j represent the row and column index of the mask image respectively, and x k and y k Together they represent the center coordinates of the kth true box, and K represents the number of true boxes whose absolute scale is smaller than a threshold ε. In this embodiment of the present invention, the threshold ε is set to 32.

[0070] After obtaining the mask map M, the cross entropy loss is used to constrain the learning of the key area prediction unit, and the key area response map prediction loss L key The calculation formula is as follows:

[0071] L key =CE(W,M)

[0072] Where CE(·) represents the cross entropy loss.

[0073] After obtaining the response map W that can reflect the distribution of key areas of small-scale pedestrians, the response map is used to enhance the features of small-scale pedestrians. Specifically, W is scaled to the same size as the fused feature F through maximum pooling, and the fused feature F is enhanced to obtain the enhanced fused feature F'. The feature enhancement process is shown as follows:

[0074]

[0075] Among them, MaxPool(·) represents the maximum pooling operation, represents element-wise matrix multiplication, Represents element-wise addition of matrices.

[0076] 5. Training a Multimodal Pedestrian Detection Network

[0077] The embodiment of the present invention adopts an end-to-end approach to train a multimodal pedestrian detection network. During the training of the multimodal pedestrian detection network, the input images of the two modalities are enhanced by random flipping and then fed into the network at the same time. After passing through the designed dual-branch feature extraction network, imaging quality perception module, multimodal feature fusion module and key area guided feature enhancement module, the enhanced fusion feature F' is obtained. Subsequently, the enhanced fusion feature F' is fed into the detection head to obtain the pedestrian prediction result. During the training of the multimodal pedestrian detection network, manually annotated pedestrian real frames are used to constrain the prediction results of the network. The overall loss L of the training process is expressed as follows:

[0078] L=L key +L reg +L cls

[0079] Among them, L key represents the prediction loss of key region response map in key region guided feature enhancement module, L reg and L cls Respectively represent the target regression loss and target classification loss in the detection head. In the embodiment of the present invention, the regression loss L in the detection head reg and classification loss L cls The generalized intersection-over-union loss and the focusing loss are used for implementation, both of which are well known in the art and will not be described in detail here.

[0080] After completing the multimodal pedestrian detection network training, the trained network parameter model is deployed on hardware such as drones and servers. The model can perform multimodal pedestrian detection using visible light and infrared multimodal images as input.

[0081] In summary, an embodiment of the present invention proposes a multimodal pedestrian detection method, which aims to effectively perceive the regional imaging quality differences between visible light and infrared images, fully integrate visible light-infrared multimodal information, and enhance the features of key areas of small-scale pedestrians to achieve accurate pedestrian detection in aerial photography scenes.

[0082] Unless otherwise specified, the embodiments of the present invention do not limit the models of the components. Any component that can perform the above functions may be used.

[0083] Those skilled in the art will understand that the accompanying drawings are only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0084] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal pedestrian detection method, characterized in that: The method comprises: The imaging quality perception module is used to perceive the differences in imaging quality between different regions of visible light and infrared images and to exploit the complementary characteristics of multimodal images. The multimodal feature fusion module uses the normalized imaging quality spatial distribution map to guide the feature fusion of visible light and infrared multimodal images to obtain a multimodal fusion feature representation; The key region guided feature enhancement module predicts the key regions where small-scale pedestrians are located and enhances the features of the key regions to obtain a more discriminative fused feature representation; The enhanced fusion features are fed into the detection head to obtain pedestrian prediction results.

2. A multimodal pedestrian detection method according to claim 1, characterized in that: The imaging quality perception module is: Visible light image I v and infrared image I t As input, each modality image is evenly divided into m×n subregions; Using a vision-language basic model, the imaging quality of each sub-region of the multimodal image is evaluated. The evaluation results of each sub-region are integrated to obtain the spatial distribution map of the imaging quality of the visible light image and the infrared image. The spatial distribution diagram of imaging quality of each modality was normalized.

3. The multimodal pedestrian detection method according to claim 2, characterized in that: The spatial distribution diagram of imaging quality is: Among them, Q v and Q t Represent the spatial distribution diagrams of imaging quality of visible light images and infrared images respectively, and Represent the imaging quality assessment results of the first sub-region of the visible light image and infrared image, respectively. and Represent the imaging quality assessment results of the m×nth sub-region of the visible light image and infrared image respectively, VL() represents the vision-language basic model, and prompt represents the language prompt.

4. The multimodal pedestrian detection method according to claim 1, characterized in that: The normalized image quality spatial distribution diagram Q' v and Q t Under the guidance of ', the two modal features are fused to give full play to the complementary advantages of the two modal information. The fusion process is as follows: Among them, F v and F t Represent the original features of the two modal images respectively, F represents the multimodal fusion feature representation, represents element-wise matrix multiplication, represents element-by-element addition of matrices, cat(·) represents feature concatenation, and conv(·) represents a convolutional layer with a convolution kernel size of 1×1.

5. The multimodal pedestrian detection method according to claim 1, characterized in that: The key area guided feature enhancement module is: For the input multimodal fusion feature representation F, first input the key area prediction unit composed of the initialization layer, the spatial transformation encoder and the channel compression layer to obtain the key area prediction response map W where the small-scale pedestrian is located; Calculate the absolute scale D of each ground truth box abs , select the real box whose absolute scale is smaller than the threshold ε and represent the key area where it is located as a Gaussian mask map; Cross entropy loss is used to constrain the learning of key region prediction units, and the response map is used to enhance the features of small-scale pedestrians.

6. The multimodal pedestrian detection method according to claim 5, characterized in that: The process of selecting a true box whose absolute scale is smaller than the threshold ε and representing the key area where it is located as a Gaussian mask is as follows: Among them, i and j represent the row and column index of the mask image respectively, and x k and y k They jointly represent the center coordinates of the kth true box, and K represents the number of true boxes whose absolute scale is smaller than the threshold ε.

7. The multimodal pedestrian detection method according to claim 5, characterized in that: The method of using the response map to enhance the features of small-scale pedestrians is as follows: scaling W to the same size as the fusion feature F through maximum pooling, and enhancing the fusion feature F to obtain the enhanced fusion feature F'. The feature enhancement process is shown as follows: Among them, MaxPool(·) represents the maximum pooling operation, represents element-wise matrix multiplication, Represents element-wise addition of matrices.

Images