A multi-modal pedestrian detection method

Abstract

The application discloses a kind of multi-modal pedestrian detection methods, the method includes: imaging quality perception module is used to perceive the imaging quality difference of different regions of visible light image and infrared image, excavates the complementary characteristics of multi-modal image;Multi-modal feature fusion module guides the feature fusion of visible light-infrared multi-modal image with normalized imaging quality space distribution diagram, obtains multi-modal fusion feature representation;Key area guided feature enhancement module is by predicting the key area where small scale pedestrian is located, and key area is enhanced to feature, obtains more discriminative fusion feature representation;Enhanced fusion feature is sent into detection head, and pedestrian prediction result is obtained.The application obtains more comprehensive visible light-infrared multi-modal feature representation, to improve the accuracy of pedestrian detection.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and pedestrian detection, and in particular to a multimodal pedestrian detection method. Background Technology

[0002] Accurate pedestrian detection in drone aerial photography can effectively improve pedestrian search efficiency in complex environments such as mountainous terrain, and is widely used in tasks such as wilderness rescue. However, real-world applications require all-day pedestrian detection, and factors such as high daytime exposure and low nighttime illumination inevitably affect the accuracy of pedestrian detection. Considering that visible light imaging has the ability to describe detailed information such as scene color and texture, while infrared imaging has thermal radiation sensing characteristics, combining visible light and infrared images can effectively achieve all-day pedestrian detection. Therefore, exploring a visible light-infrared joint multimodal pedestrian detection method, and obtaining accurate pedestrian detection results by fully integrating multimodal complementary information, has significant research significance and application value.

[0003] In recent years, researchers have proposed a series of visible-infrared multimodal pedestrian detection methods, achieving high-quality pedestrian detection by effectively fusing multimodal information. Zhang et al. proposed a pedestrian detection method based on overall illumination condition perception, which fuses multimodal information under different illumination conditions by sensing image-level illumination changes to improve pedestrian detection performance. Chen et al. proposed an attention alignment network, which effectively detects pedestrian targets in the scene by making the network pay more attention to pedestrian regions and mitigating the impact of environmental illumination imbalance on multimodal information fusion.

[0004] Most of the methods described above assess the overall imaging quality difference between visible light and infrared images by sensing the overall illumination conditions of images from different modalities, thereby achieving visible light-infrared feature fusion. However, in aerial images, due to the wide coverage of the scene and the complexity of the background, there is also an imbalance in imaging quality between different regions of the visible light and infrared images. Furthermore, pedestrian targets in aerial scenes are relatively small in scale, making small-scale pedestrian features susceptible to interference from complex background features during image feature extraction. This affects the network's effective detection of pedestrian targets and limits further improvements in pedestrian detection performance. Summary of the Invention

[0005] This invention provides a multimodal pedestrian detection method. By sensing regional imaging quality differences in multimodal images and enhancing features in key regions containing small-scale targets, this invention fully explores the complementary characteristics of multimodal imaging and enhances key region features. This mitigates the adverse effects of visible light-infrared imaging quality imbalance and small pedestrian scale in aerial photography scenes, obtaining a more comprehensive visible light-infrared multimodal feature representation, thereby improving the accuracy of pedestrian detection. See the description below for details:

[0006] A multimodal pedestrian detection method, the method comprising:

[0007] The imaging quality perception module is used to perceive the differences in imaging quality between different regions of visible light and infrared images, and to explore the complementary characteristics of multimodal images.

[0008] The multimodal feature fusion module uses the normalized imaging quality spatial distribution map to guide feature fusion of visible light-infrared multimodal images, thereby obtaining multimodal fused feature representations.

[0009] The key region-guided feature enhancement module predicts the key regions where pedestrians are located at a small scale and enhances the features of these key regions to obtain a more discriminative fused feature representation.

[0010] The enhanced fusion features are fed into the detection head to obtain pedestrian prediction results.

[0011] The imaging quality sensing module is as follows:

[0012] Visible light image I v and infrared image I t As input, each modal image is divided into an average of m×n sub-regions;

[0013] A vision-language-based model is used to assess the imaging quality of each sub-region of a multimodal image; the assessment results of each sub-region are integrated to obtain a spatial distribution map of the imaging quality of visible light and infrared images.

[0014] Normalize the spatial distribution of imaging quality for each modality.

[0015] The spatial distribution map of imaging quality is as follows:

[0016]

[0017] Among them, Q v and Q t These represent the spatial distribution maps of imaging quality for visible light and infrared images, respectively. and These represent the imaging quality assessment results for the first sub-region of the visible light image and the infrared image, respectively. and represents the imaging quality assessment results of the m×n sub-region of the visible light image and the infrared image, respectively. VL() represents the visual-language basic model, and prompt represents the language prompt.

[0018] Among them, the normalized spatial distribution map of imaging quality Q' v and Q tUnder the guidance of [the relevant authority], the features of the two modalities are integrated to leverage the complementary advantages of the two modalities. The integration process is as follows:

[0019]

[0020] Among them, F v and F t Let F represent the original features of the two modal images, and let F represent the multimodal fusion feature representation. This represents element-wise matrix multiplication. This indicates that the matrix is ​​added element by element, cat(·) indicates feature concatenation, and conv(·) indicates a convolutional layer with a kernel size of 1×1.

[0021] The feature enhancement module guided by the key region is as follows:

[0022] For the input multimodal fusion feature representation F, the key region prediction unit consisting of an initialization layer, a spatial transformation encoder, and a channel compression layer is first input to obtain the key region prediction response map W where the pedestrian is located at a small scale.

[0023] Calculate the absolute scale D of each ground truth bounding box. abs Select the real bounding boxes with an absolute scale smaller than the threshold ε and represent the key regions they contain as Gaussian mask maps;

[0024] Cross-entropy loss is used to constrain the learning of prediction units in key regions, and response maps are used to enhance the features of pedestrians at small scales.

[0025] The process of selecting ground truth bounding boxes with an absolute scale smaller than the threshold ε and representing their key regions as Gaussian mask images is as follows:

[0026]

[0027] Where i and j represent the row and column indices of the mask image, respectively, and x k and y k Together they represent the center coordinates of the k-th ground truth bounding box, where K represents the number of ground truth bounding boxes with an absolute scale less than the threshold ε.

[0028] The feature enhancement of small-scale pedestrians using the response map involves scaling W to the same size as the fused feature F using max pooling, and then enhancing the fused feature F to obtain the enhanced fused feature F'. The feature enhancement process is as follows:

[0029]

[0030] Where MaxPool(·) represents the max pooling operation. This represents element-wise matrix multiplication. This indicates that the matrix is ​​added element by element.

[0031] The beneficial effects of the technical solution provided by this invention are:

[0032] 1. This invention obtains accurate pedestrian detection results in aerial photography scenes by effectively fusing visible light-infrared multimodal complementary information and enhancing key region features;

[0033] 2. This invention designs an imaging quality sensing module, which effectively mines the complementary characteristics of multimodal images by sensing the differences in imaging quality in different regions of visible light images and infrared images;

[0034] 3. This invention designs a key region-guided feature enhancement module, which predicts the key region where pedestrians are located at a small scale and effectively enhances the features of the key region to obtain a more discriminative feature representation. Attached Figure Description

[0035] Figure 1 This is a flowchart of a multimodal pedestrian detection method. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0037] I. Constructing a dual-branch feature extraction network

[0038] To obtain feature representations of visible-infrared multimodal images, this embodiment of the invention constructs a dual-branch feature extraction network.

[0039] Specifically, the paired visible light image and infrared image are represented as I. v and I t Subsequently, a dual-branch network with the same structure was used to extract the visible light image I. v and infrared image I t The feature representation and process are as follows:

[0040] F v =Backbone v (I v )

[0041] F t =Backbone t (I t )

[0042] Backbone v () and Backbone t () represent the feature extraction networks for the visible light mode branch and the infrared mode branch, respectively. v and F tThese represent visible light image features and infrared image features, respectively. In this embodiment of the invention, ResNet50 is used as the backbone network, and the parameters of the backbone network are initialized using a model pre-trained on the ImageNet dataset. Both the ResNet50 backbone network and the ImageNet dataset are well-known in the art and will not be described in detail here.

[0043] II. Design of Imaging Quality Sensing Module

[0044] In order to measure the imaging quality of different regions of a multimodal image and thus guide the subsequent multimodal feature fusion process, this embodiment of the invention designs an imaging quality perception module. By perceiving the differences in imaging quality between different regions of visible light images and infrared images, the complementary characteristics of multimodal images can be effectively mined.

[0045] Specifically, the imaging quality perception module uses visible light images I v and infrared image I t As input, firstly, the visible light image I is processed in the same way. v and infrared image I t The average region is divided into m×n sub-regions for each modality image, as shown in the following process:

[0046]

[0047] in, and These represent the first sub-regions of the visible light image and the infrared image, respectively, and so on. and These represent the m×n-th sub-regions of the visible light image and the infrared image, respectively. In this embodiment of the invention, both m and m are set to 5.

[0048] Subsequently, to obtain the imaging quality distribution of the multimodal images, a vision-language basic model was used. By constructing corresponding language prompts, the imaging quality of each sub-region of the multimodal image was evaluated. Then, the evaluation results of each sub-region were integrated to obtain the spatial distribution map of imaging quality for the visible light and infrared images. The process is shown below:

[0049]

[0050] Among them, Q v and Q t These represent the spatial distribution maps of imaging quality for visible light and infrared images, respectively. and These represent the imaging quality assessment results for the first sub-region of the visible light image and the infrared image, respectively, and so on. and These represent the imaging quality assessment results of the m×n sub-regions of the visible light image and the infrared image, respectively. VL() represents the visual-language basic model, and prompt represents the language prompt. In this embodiment of the invention, CLIP is used as the visual-language basic model, and the language prompt is set to "high-quality, appropriately bright, and clear image".

[0051] In obtaining the spatial distribution map of imaging quality Q for visible light and infrared images v and Q t Next, the spatial distribution of imaging quality for each modality is normalized. The formula for normalization is shown below:

[0052]

[0053] Among them, Q' v and Q t 'These represent the spatial distribution maps of the imaging quality after normalization of the two modes.'

[0054] III. Design of a multimodal feature fusion module

[0055] To effectively fuse complementary information from visible light and infrared multimodal images, this invention proposes a multimodal feature fusion module. The module uses a normalized spatial distribution map of imaging quality to guide feature fusion of visible light and infrared multimodal images, thereby obtaining a multimodal fused feature representation.

[0056] Specifically, in the normalized spatial distribution map of image quality Q' v and Q t Under the guidance of [the relevant authority / guideline], the features of the two modalities are effectively fused to fully leverage the complementary advantages of the two modalities. The fusion process is represented as follows:

[0057]

[0058] Among them, F v and F t Let F represent the original features of the two modal images, and let F represent the multimodal fusion feature representation. This represents element-wise matrix multiplication. This indicates that the matrix is ​​added element by element, cat(·) indicates feature concatenation, and conv(·) indicates a convolutional layer with a kernel size of 1×1.

[0059] IV. Design of Feature Enhancement Modules for Key Area Guidance

[0060] To effectively enhance the features of key areas where small-scale pedestrians are located, this embodiment of the invention designs a key area-guided feature enhancement module. By predicting the key areas where small-scale pedestrians are located and enhancing the features of the key areas, a more discriminative fusion feature representation can be obtained.

[0061] For the input multimodal fusion feature representation F, the key region prediction unit, consisting of an initialization layer, a spatial transformation encoder, and a channel compression layer, is first input to obtain the key region prediction response map W where the pedestrian is located at a small scale, as expressed by the formula:

[0062] W = sig(ch(sw(init(F))))

[0063] Here, `init(·)` represents the initialization layer, which consists of convolutional layers with a kernel size of 3×3. `sw(·)` represents the spatial transform encoder, which contains N spatial transform blocks, where... This represents the ratio of the original image size to the feature map size. Each spatial transformation block consists of a 3×3 convolutional layer, a batch normalization layer, a ReLU activation layer, and an upsampling layer, where the upsampling layer is implemented using bilinear interpolation. `ch(·)` represents a channel compression layer, composed of a 3×3 convolutional layer with 1 output channel. `sig(·)` represents the sigmoid activation function, used to map the value of W to the range (0,1).

[0064] Subsequently, to ensure that the response map effectively reflects the distribution of key pedestrian regions at a small scale, constraints are constructed using ground truth bounding boxes of small-scale pedestrian targets. Specifically, the absolute scale D of each ground truth bounding box is first calculated. abs The process is as follows:

[0065]

[0066] Where w and h are the width and height of the pedestrian's true bounding box, respectively, both in pixels.

[0067] After obtaining the absolute scale of each ground truth bounding box, using this as a metric, ground truth bounding boxes with an absolute scale smaller than a threshold ε are selected, and their key regions are represented as Gaussian mask maps M. The process is as follows:

[0068]

[0069] Where i and j represent the row and column indices of the mask image, respectively, and x k and y k The coordinates of the k-th ground truth bounding box are collectively represented by K, where K represents the number of ground truth bounding boxes with an absolute scale smaller than the threshold ε. In this embodiment of the invention, the threshold ε is set to 32.

[0070] After obtaining the mask map M, cross-entropy loss is used to constrain the learning of the prediction units in the key regions, and the prediction loss of the key region response map is L. key The calculation formula is expressed as follows:

[0071] L key =CE(W,M)

[0072] Where CE(·) represents the cross-entropy loss.

[0073] After obtaining the response map W, which reflects the distribution of key pedestrian areas at a small scale, the features of small-scale pedestrians are enhanced using the response map. Specifically, W is scaled to the same size as the fused feature F through max pooling, and the fused feature F is enhanced to obtain the enhanced fused feature F'. The feature enhancement process is represented as follows:

[0074]

[0075] Where MaxPool(·) represents the max pooling operation. This represents element-wise matrix multiplication. This indicates that the matrix is ​​added element by element.

[0076] V. Training the Multimodal Pedestrian Detection Network

[0077] This invention employs an end-to-end approach to train a multimodal pedestrian detection network. During training, the input images from both modalities are enhanced by random flipping and then simultaneously fed into the network. After passing through a designed dual-branch feature extraction network, an image quality perception module, a multimodal feature fusion module, and a key region-guided feature enhancement module, the enhanced fused feature F' is obtained. Subsequently, the enhanced fused feature F' is fed into the detection head to obtain the pedestrian prediction result. During the training of the multimodal pedestrian detection network, manually annotated ground truth bounding boxes are used to constrain the network's prediction results. The overall loss L during the training process is expressed as follows:

[0078] L = L key +L reg +L cls

[0079] Among them, L key L represents the prediction loss of the key region response map in the key region-guided feature enhancement module. reg and L cls Let L represent the target regression loss and target classification loss in the detection head, respectively. In this embodiment of the invention, the regression loss L in the detection head... reg And classification loss L cls The generalized intersection-union loss and the focus loss are respectively used, both of which are well known in the field and will not be described in detail here.

[0080] After training the multimodal pedestrian detection network, the trained network parameter model is deployed on hardware such as drones and servers. This model can perform multimodal pedestrian detection using visible light-infrared multimodal images as input.

[0081] In summary, this invention proposes a multimodal pedestrian detection method. This method aims to effectively perceive the regional imaging quality differences between visible light and infrared images, fully integrate visible light-infrared multimodal information, and enhance the features of key areas of small-scale pedestrians to achieve accurate pedestrian detection in aerial photography scenes.

[0082] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0083] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0084] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal pedestrian detection method, characterized in that, The method includes: The imaging quality perception module is used to perceive the differences in imaging quality between different regions of visible light and infrared images, and to explore the complementary characteristics of multimodal images. The multimodal feature fusion module uses the normalized imaging quality spatial distribution map to guide feature fusion of visible light-infrared multimodal images, thereby obtaining multimodal fused feature representations. The key region-guided feature enhancement module predicts the key regions where pedestrians are located at a small scale and enhances the features of these key regions to obtain a more discriminative fused feature representation. The enhanced fusion features are fed into the detection head to obtain pedestrian prediction results; Normalized image quality spatial distribution map and Under the guidance of [the relevant authority], the features of the two modalities are integrated to leverage the complementary advantages of the two modalities. The integration process is as follows: ； in, and These represent the original features of the two modal images, This represents the multimodal fusion feature representation. This represents element-wise matrix multiplication. This indicates that the matrix is ​​added element by element. Indicates feature cascading, This indicates a convolutional layer with a kernel size of 1×1; The feature enhancement module guided by the key region is as follows: For the input multimodal fusion feature representation First, a key region prediction unit consisting of an initialization layer, a spatial transformation encoder, and a channel compression layer is input to obtain a small-scale prediction response map of the key region where pedestrians are located. ; Calculate the absolute scale of each ground truth bounding box. Select an absolute scale smaller than the threshold. The true bounding box is represented as a Gaussian mask map of the key region it contains; Cross-entropy loss is used to constrain the learning of prediction units in key regions, and response maps are used to enhance the features of pedestrians at small scales. Wherein, the selection of an absolute scale less than a threshold The process of defining the true bounding box and representing the key region it contains as a Gaussian mask is as follows: ； in, and These represent the row and column indices of the mask image, respectively. and Jointly representing the first The center coordinates of the real bounding box This indicates that the absolute scale is less than the threshold. The actual number of frames.

2. The multimodal pedestrian detection method according to claim 1, characterized in that, The imaging quality sensing module is: Visible light images and infrared images As input, each modality image is averaged into Sub-regions; A vision-language-based model is used to assess the imaging quality of each sub-region of a multimodal image; the assessment results of each sub-region are integrated to obtain a spatial distribution map of the imaging quality of visible light and infrared images. Normalize the spatial distribution of imaging quality for each modality.

3. The multimodal pedestrian detection method according to claim 2, characterized in that, The spatial distribution map of image quality is as follows: ； ； in, and These represent the spatial distribution maps of imaging quality for visible light and infrared images, respectively. and These represent the imaging quality assessment results for the first sub-region of the visible light image and the infrared image, respectively. and These represent the visible light image and the infrared image, respectively. Imaging quality assessment results for each sub-region Represents the visual-language basic model, Indicates language prompts.

4. The multimodal pedestrian detection method according to claim 1, characterized in that, The method of enhancing the features of small-scale pedestrians using response maps involves: using max pooling... Scaling to fused features For the same size, the fused features Enhancement is performed to obtain enhanced fusion features. The feature enhancement process is represented as follows: ； in, This indicates a max pooling operation. This represents element-wise matrix multiplication. This indicates that the matrix is ​​added element by element.

Images