A cross-modal pedestrian detection method based on Gaussian cross-attention network

Abstract

The present invention provides a cross-modal pedestrian detection method based on a Gaussian cross-attention network, comprising: S1, obtaining multi-source pedestrian images; S2, performing data preprocessing on the multi-source pedestrian images to obtain a multi-source pedestrian image training set; S3, manually annotating the multi-source pedestrian images to obtain true pedestrian position bounding box coordinates; S4, constructing a detection network based on Gaussian cross-attention; S5, using the multi-source pedestrian image training set to train the detection network based on Gaussian cross-attention to obtain a trained detection network based on Gaussian cross-attention; S6, using the trained detection network based on Gaussian cross-attention to perform pedestrian detection. The present invention uses a detection network based on Gaussian cross-attention to extract deep significant features of multi-source pedestrian images and fuse them, directly obtaining bounding box coordinates representing pedestrian positions, which is more convenient and flexible in network training and practical use, saving computing resources and reducing time consumption.

Description

Technical Field

[0001] The present invention relates to the technical field of pedestrian detection, and in particular to a cross-modal pedestrian detection method based on a Gaussian cross attention network. Background Art

[0002] Pedestrian detection is an important research area with wide applications in autonomous driving, surveillance, and robotics. Safety assurance has been improved in recent years by relying on the development of detection algorithms. Pedestrian detection tasks have achieved good results in detecting visible images and videos. However, there are still some challenges in visible images such as low resolution, occlusion, power contrast, and poor lighting conditions that limit the accuracy of pedestrian detection.

[0003] Thermal imaging eliminates the limitations of visible light images in conditions such as low light and bad weather. Thermal imaging cameras detect objects based on infrared radiation, and there is a significant thermal difference between people and their surroundings, so pedestrians can be easily identified. However, during the day, it is difficult to distinguish between people and interfering objects because the background information is similar in temperature to the pedestrians in the thermal image. This leads to the conclusion that thermal images are more suitable for nighttime, while visible light images are more suitable for pedestrian detection during the day.

[0004] In the existing multi-source pedestrian image detection methods, CNN-based fusion strategies enhance the expression of pedestrian features in images by learning local features, but they lack the ability to extract long-range dependencies from images, which leads to the loss of some basic global context that may be useful for pedestrian detection. Therefore, combining local features with long-range dependency density can increase global context information, which helps to enhance the fusion of significant features of multi-source images, thereby further improving the accuracy of detection. End-to-end neural networks can integrate feature extraction and visual tasks, which is more convenient and flexible in network training and practical application. Applying them to multi-source pedestrian image detection is of great significance and value. Summary of the invention

[0005] The present invention aims to provide a cross-modal pedestrian detection method based on Gaussian cross attention network to solve the problems existing in the above-mentioned existing multi-source pedestrian image detection methods.

[0006] The present invention provides a cross-modal pedestrian detection method based on Gaussian cross attention network, comprising the following steps:

[0007] S1, obtain multi-source pedestrian images;

[0008] S2, performing data preprocessing on the multi-source pedestrian images obtained in step S1 to obtain a multi-source pedestrian image training set;

[0009] S3, manually annotating the multi-source pedestrian images obtained in step S1 to obtain the coordinates of the true pedestrian position bounding box;

[0010] S4, constructs a detection network based on Gaussian cross attention;

[0011] S5, using the multi-source pedestrian image training set obtained in step S2 to train the Gaussian cross-attention based detection network constructed in step S4, during the training process, using a loss function to calculate the difference between the predicted pedestrian position bounding box coordinates output by the Gaussian cross-attention based detection network and the true pedestrian position bounding box coordinates obtained in step S3, and optimizing the parameters of the detection network through a back-propagation algorithm, thereby obtaining a trained Gaussian cross-attention based detection network;

[0012] S6, pedestrian detection using the trained Gaussian cross-attention based detection network.

[0013] In some embodiments, the multi-source pedestrian image includes a visible light pedestrian image and an infrared light pedestrian image.

[0014] In some embodiments, the method for performing data preprocessing on the multi-source pedestrian images obtained in step S1 in step S2 is:

[0015] The multi-source pedestrian images are normalized by calculating the mean and variance of the overall data to obtain a multi-source pedestrian image training set.

[0016] In some embodiments, the Gaussian cross-attention based detection network constructed in step S4 includes: a feature extraction network, a cross-attention based feature fusion network, and a prediction network;

[0017] The feature extraction network is a series of convolutional neural networks, which are used to extract visible light features and infrared light features from visible light pedestrian images and infrared light pedestrian images respectively;

[0018] The feature fusion network based on cross attention includes a self-attention mechanism module, a learnable Gaussian attention module and a cross-attention fusion module; the self-attention mechanism module reduces the dimension of the infrared feature map through a 1×1 convolution layer to obtain an index feature map Q; the learnable Gaussian attention module reduces the dimension of the visible light feature map through a 1×1 convolution layer and filters the visible light background features through a learnable Gaussian attention layer to obtain a key-value feature map K; the similarity between the index feature map Q and the key-value feature map K is calculated through a parameter operation layer and an attention heat map containing a significant feature weight distribution is obtained through a Softmax function; the cross-attention fusion module uses a 1×1 convolution layer to reduce the dimension of the visible light feature map to obtain a content feature map V, and aggregates the content feature map V with the attention heat map and then adds it to the visible light feature map to obtain a fused feature map;

[0019] The prediction network includes a target classification branch and a bounding box regression branch respectively composed of multiple convolutional layers, fully connected layers and softmax layers, and is used to obtain the coordinates of the predicted pedestrian position bounding box according to the fused feature map.

[0020] In some embodiments, the loss function is a smooth L1 loss function.

[0021] In some embodiments, the optimization algorithm used in step S5 when training the detection network based on Gaussian cross attention is a stochastic gradient descent optimization algorithm.

[0022] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0023] 1. The present invention adopts a detection network based on Gaussian cross attention to extract and fuse deep significant features of multi-source pedestrian images, directly obtaining the bounding box coordinates representing the pedestrian position, which is more convenient and flexible in network training and practical application, saving computing resources and reducing time consumption.

[0024] 2. The detection network based on Gaussian cross-attention constructed by the present invention integrates a feature extraction network, a feature fusion network based on cross-attention, and a prediction network, fully extracts and fuses the deep significant features of multi-source pedestrian images, improves the robustness of the features, and can obtain a detection model with stronger performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 This is a schematic diagram of a cross-modal pedestrian detection method based on a Gaussian cross-attention network according to an embodiment of the present invention.

[0027] Figure 2 It is a structural diagram of a detection network based on Gaussian cross attention according to an embodiment of the present invention.

[0028] Figure 3 It is a structural diagram of a feature fusion network based on cross attention according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0030] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] Example

[0032] like Figure 1 As shown, this embodiment proposes a cross-modal pedestrian detection method based on a Gaussian cross attention network, comprising the following steps:

[0033] S1, obtain a multi-source pedestrian image; in this embodiment, the multi-source pedestrian image includes a visible light pedestrian image X0∈N 800×600×3 and infrared pedestrian image X1∈N 800×600×3 ; Among them, 800 and 600 represent the image size, and 3 represents the dimension (that is, the number of input image channels). This value is only an example of this embodiment and should not be limited to it.

[0034] S2, performing data preprocessing on the multi-source pedestrian images obtained in step S1 to obtain a multi-source pedestrian image training set; the data preprocessing method is:

[0035] The multi-source pedestrian images are normalized by calculating the mean and variance of the overall data to obtain a multi-source pedestrian image training set, including the visible light pedestrian image X'0∈R after data preprocessing 800×600×3 And the infrared pedestrian image X'1∈R after data preprocessing 800×600×3 .

[0036] In addition, data enhancement may be performed on the multi-source pedestrian images before normalization, for example, the multi-source pedestrian images may be horizontally flipped and rotated at a small angle with a probability of 50%.

[0037] S3, manually annotating the multi-source pedestrian images obtained in step S1 to obtain the true value pedestrian position bounding box coordinates (x1, y1, x2, y2);

[0038] S4, constructs a detection network based on Gaussian cross attention; Figure 2As shown, in this embodiment, the detection network based on Gaussian cross attention is constructed including: a feature extraction network, a feature fusion network based on cross attention, and a prediction network;

[0039] The feature extraction network is a series of convolutional neural networks, which are used to extract visible light features and infrared light features from visible light pedestrian images and infrared light pedestrian images respectively;

[0040] The cross-attention-based feature fusion network includes a self-attention mechanism module, a learnable Gaussian attention module and a cross-attention fusion module; Figure 3 As shown in FIG. 1 , the self-attention mechanism module reduces the dimension of the infrared feature map through a 1×1 convolution layer to obtain an index feature map Q; the learnable Gaussian attention module reduces the dimension of the visible light feature map through a 1×1 convolution layer and filters the visible light background features through a learnable Gaussian attention layer to obtain a key-value feature map K; the similarity between the index feature map Q and the key-value feature map K is calculated through the parameter operation layer and the attention heat map containing the weight distribution of significant features is obtained through the Softmax function; the cross-attention fusion module uses a 1×1 convolution layer to reduce the dimension of the visible light feature map to obtain a content feature map V, and aggregates the content feature map V with the attention heat map and then adds it to the visible light feature map to obtain a fused feature map F∈R 32×25×2048 , to fuse the target’s significant depth features in the infrared light feature map and the visible light feature map;

[0041] The prediction network includes a target classification branch and a bounding box regression branch respectively composed of multiple convolutional layers, fully connected layers and softmax layers, and is used to obtain the predicted pedestrian position bounding box coordinates (x'1, y'1, y'2, y'2) according to the fused feature map.

[0042] S5, use the multi-source pedestrian image training set obtained in step S2 to train the detection network based on Gaussian cross attention constructed in step S4. During the training process, a loss function is used to calculate the difference between the predicted pedestrian position bounding box coordinates (x'1, y'1, y'2, y'2) output by the detection network based on Gaussian cross attention and the true pedestrian position bounding box coordinates (x1, y1, x2, y2) obtained in step S3, and the parameters of the detection network are optimized by the back propagation algorithm to obtain a trained detection network based on Gaussian cross attention. In this embodiment, the training parameters can be set to: a total of 12 epochs are trained, the batch size is set to 16, the initial learning rate is 0.0015, the attenuation coefficient of the learning rate in the 8th and 11th epochs is 0.1, and the optimization algorithm used in training is the stochastic gradient descent optimization algorithm (SGD algorithm).

[0043] S6, using the trained Gaussian cross attention based detection network to detect pedestrians. Similarly, obtain a multi-source pedestrian image to be detected, perform the same data preprocessing as step S2 on the multi-source pedestrian image to be detected, and then input it into the trained Gaussian cross attention based detection network to detect pedestrians.

[0044] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A cross-modal pedestrian detection method based on Gaussian cross attention network, characterized in that: The steps include: S1, obtain multi-source pedestrian images; S2, performing data preprocessing on the multi-source pedestrian images obtained in step S1 to obtain a multi-source pedestrian image training set; S3, manually annotating the multi-source pedestrian images obtained in step S1 to obtain the coordinates of the true pedestrian position bounding box; S4, constructs a detection network based on Gaussian cross attention; S5, using the multi-source pedestrian image training set obtained in step S2 to train the Gaussian cross-attention based detection network constructed in step S4, during the training process, using a loss function to calculate the difference between the predicted pedestrian position bounding box coordinates output by the Gaussian cross-attention based detection network and the true pedestrian position bounding box coordinates obtained in step S3, and optimizing the parameters of the detection network through a back-propagation algorithm, thereby obtaining a trained Gaussian cross-attention based detection network; S6, pedestrian detection using the trained Gaussian cross-attention based detection network; The detection network based on Gaussian cross attention constructed in step S4 includes: a feature extraction network, a feature fusion network based on cross attention, and a prediction network; The feature extraction network is a series of convolutional neural networks, which are used to extract visible light features and infrared light features from visible light pedestrian images and infrared light pedestrian images respectively; The feature fusion network based on cross attention includes a self-attention mechanism module, a learnable Gaussian attention module and a cross-attention fusion module; the self-attention mechanism module reduces the dimension of the infrared feature map through a 1×1 convolution layer to obtain an index feature map Q; the learnable Gaussian attention module reduces the dimension of the visible light feature map through a 1×1 convolution layer and filters the visible light background features through a learnable Gaussian attention layer to obtain a key-value feature map K; the similarity between the index feature map Q and the key-value feature map K is calculated through a parameter operation layer and an attention heat map containing a significant feature weight distribution is obtained through a Softmax function; the cross-attention fusion module uses a 1×1 convolution layer to reduce the dimension of the visible light feature map to obtain a content feature map V, and aggregates the content feature map V with the attention heat map and then adds it to the visible light feature map to obtain a fused feature map; The prediction network includes a target classification branch and a bounding box regression branch respectively composed of multiple convolutional layers, fully connected layers and softmax layers, and is used to obtain the coordinates of the predicted pedestrian position bounding box according to the fused feature map.

2. The cross-modal pedestrian detection method based on Gaussian cross attention network according to claim 1 is characterized in that: The multi-source pedestrian image includes a visible light pedestrian image and an infrared light pedestrian image.

3. The cross-modal pedestrian detection method based on Gaussian cross attention network according to claim 2 is characterized in that: The method for performing data preprocessing on the multi-source pedestrian images obtained in step S1 in step S2 is: The multi-source pedestrian images are normalized by calculating the mean and variance of the overall data to obtain a multi-source pedestrian image training set.

4. The cross-modal pedestrian detection method based on Gaussian cross attention network according to claim 1, characterized in that: The loss function is a smooth L1 loss function.

5. The cross-modal pedestrian detection method based on Gaussian cross attention network according to claim 1, characterized in that: The optimization algorithm used in step S5 when training the detection network based on Gaussian cross attention is the stochastic gradient descent optimization algorithm.

Images