Target detection method and device based on cross attention multi-scale fusion and medium

Abstract

The invention discloses a target detection method and device based on cross attention multi-scale fusion and a medium, and relates to the field of target detection, and the method comprises the steps: obtaining a visible light image and an infrared image of a to-be-detected target, and carrying out the target detection through employing a trained cross attention multi-scale fusion detection network; wherein the cross attention multi-scale fusion detection network comprises a backbone network, a background sensing module, a cross attention fusion module and a detection head; the backbone network is used for extracting features of different scales; the background sensing module is used for predicting illumination and contrast; the cross attention fusion module is used for performing cross fusion on the features extracted by the backbone network by adopting differential attention and homogeneous attention according to the illumination and contrast predicted by the background sensing module to obtain a multi-modal fusion feature map; and the detection head is used for determining the position and category of the to-be-detected target according to the multi-modal fusion feature map. According to the invention, the accuracy of target detection is improved.

Description

Technical Field

[0001] The present application relates to the field of target detection, and in particular to a target detection method, device and medium based on cross-attention multi-scale fusion. Background Art

[0002] Due to its wide application in autonomous driving, search and rescue, patrol and surveillance, etc., object detection has always been a core technology in computer vision tasks. However, limited by the sensor's perception performance and adaptive capabilities, work based on a single modality often has difficulty in dealing with sensor failures and dynamic disturbances in the environment, and lacks robustness.

[0003] In order to alleviate the limitations of single modality in imaging and achieve all-weather monitoring, more and more researchers have begun to pay attention to multispectral target detection, which achieves better detection accuracy by fusing visual information of different modalities. However, most existing methods use a simple fusion mechanism, which is unable to learn complementary information between modalities and lacks the guidance of prior knowledge, resulting in low accuracy of target detection results. Summary of the invention

[0004] The purpose of this application is to provide a target detection method, device and medium based on cross-attention multi-scale fusion, which can improve the accuracy of target detection.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a target detection method based on cross-attention multi-scale fusion, comprising:

[0007] Acquire visible light images and infrared images of the target to be detected;

[0008] According to the visible light image and the infrared image, target detection is performed using a trained cross-attention multi-scale fusion detection network to determine the position and category of the target to be detected;

[0009] Among them, the cross-attention multi-scale fusion detection network includes a backbone network, a background perception module, a cross-attention fusion module and a detection head; the backbone network is used to extract features of different scales of the visible light image and the infrared image; the background perception module is used to predict the illumination and contrast of the visible light image and the infrared image; the cross-attention fusion module is used to cross-fuse the features extracted by the backbone network according to the illumination and contrast predicted by the background perception module, using differential attention and homogeneous attention to obtain a multimodal fusion feature map; the detection head is used to determine the position and category of the target to be detected according to the multimodal fusion feature map.

[0010] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned target detection method based on cross-attention multi-scale fusion.

[0011] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned target detection method based on cross-attention multi-scale fusion.

[0012] According to the specific embodiments provided in this application, this application has the following technical effects:

[0013] The present application provides a target detection method, device and medium based on cross-attention multi-scale fusion, adopts a cross-attention multi-scale fusion detection network to realize the adaptive fusion of visible light and infrared features, uses a background perception module to predict illumination and contrast, inputs the feature map extracted by the backbone network into a cross-attention fusion module composed of differential attention and homogeneous attention, homogeneous attention focuses on shared features within the modality to avoid introducing a large number of redundant features, differential attention enhances the difference features between modalities to promote the fusion of complementary information, so as to enhance the inherent and discriminative features within and between modalities, cross-fuse the enhanced features using the illumination and contrast predicted by the background perception module, determines the position and category of the target to be detected according to the multimodal fusion feature map, and improves the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 is a schematic diagram of a visible light image in a low illumination scene;

[0016] Figure 2 for Figure 1 Detection result diagram of

[0017] Figure 3 Schematic diagram of an infrared image in a low-light scene;

[0018] Figure 4 for Figure 3 Detection result diagram of

[0019] Figure 5 is a schematic diagram of a visible light image under a normal lighting scene;

[0020] Figure 6 for Figure 5 Detection result diagram of

[0021] Figure 7 is a schematic diagram of an infrared image under normal lighting scene;

[0022] Figure 8 for Figure 7 Detection result diagram of

[0023] Fig. 9 A flowchart of a target detection method based on cross-attention multi-scale fusion provided in one embodiment of the present application;

[0024] Fig.10 A schematic diagram of the structure of a cross-attention multi-scale fusion detection network provided in an embodiment of the present application;

[0025] Fig.11 A schematic diagram of a background perception module provided in an embodiment of the present application;

[0026] Fig.12 is a schematic diagram of the CSP structure;

[0027] Fig.13 A schematic diagram of the structure of differential attention provided in one embodiment of the present application;

[0028] Fig.14 A schematic diagram of the structure of homogeneous attention provided in one embodiment of the present application;

[0029] Fig.15 Schematic diagram of the distribution and scale changes of the width of the three data sets;

[0030] Fig.16 Schematic diagram of the height distribution and scale changes of the three data sets;

[0031] Fig.17 A schematic diagram of a detection result of the VEDAI dataset;

[0032] Fig.18 Another schematic diagram of the detection results of the VEDAI dataset;

[0033] Fig.19 This is a schematic diagram of the detection results of the daytime scene in the FLIR dataset;

[0034] Fig. 20 This is a schematic diagram of the detection results of night scenes in the FLIR dataset;

[0035] Fig.21 A schematic diagram of a detection result of the LLVIP dataset;

[0036] Fig. 22 Another schematic diagram of the detection results of the LLVIP dataset. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0038] Visible light sensors can provide rich color and texture information, but their images are easily affected by different objects with the same spectrum or different spectra of the same object, especially under complex conditions such as lighting changes, rain, fog, etc., and the contrast of some targets is poor. In contrast, infrared sensors are more sensitive to temperature and radiation, and perform well under the above harsh conditions, but infrared images have low resolution, unclear edges, and it is difficult to capture detailed visual features. Figures 1 to 4 As shown in the figure, in low-light scenes, the visible light spectrum has poor hierarchy, and the foreground and background cannot be distinguished either in the image or in the feature space. In contrast, the target in the infrared image can be clearly distinguished from the background. The features of the target and the background are also separated into two groups. Figures 5 to 8 Even with the great progress of deep learning technology, detection technology using only a single sensor will still encounter difficulties in increasingly complex environments.

[0039] In order to alleviate the limitations of single modality in imaging and achieve all-weather monitoring, more and more researchers have begun to pay attention to multispectral target detection. By fusing visual information from different spectra, higher detection accuracy can be achieved. The solutions for multimodal target detection are currently divided into manual methods and deep learning methods. On the one hand, the manual method is implemented through the traditional "feature extraction + classifier" method, but the feature extraction ability of the manually designed multimodal operator is limited, and it is difficult to obtain reliable detection results. On the other hand, due to its strong representation ability, feature fusion methods based on convolutional neural networks (CNNs) have been widely used in multimodal target detection, most of which are based on two-stream convolutional neural networks. However, the fusion strategies involved in two-stream convolutional neural networks mostly use simple element addition, multiplication and splicing. Although they have higher performance than single-modality detection, they do not fully consider cross-modal fusion and interaction, and cannot utilize the complementary information between different modalities, which means that the network has poor adaptability when dealing with high-similarity targets. Worse, these simple fusion mechanisms lack long-term dependence, which may aggravate the imbalance of the network, resulting in unsatisfactory detection results. In addition, considering the differences in visible light and infrared imaging methods and their sensitivity to light, some works have proposed illumination-aware networks to help the network learn the weight ratios of different modalities. The main idea of ​​these methods is to predict the illumination-aware weights of the input image through a formulated gate function, and then the two feature extraction branches are weighted and fused in proportion to obtain the final detection result. However, it is not enough to use only illumination information as prior knowledge to calculate modal weights, because this method cannot measure other relevant factors, nor can it achieve satisfactory corrections for dawn, dusk, strong light or dim scenes. Therefore, the goal of this application is to improve the accuracy of target detection results by designing an effective fusion mechanism and learning reliable prior knowledge to make full use of the inherent and complementary information within and between modalities.

[0040] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0041] In an exemplary embodiment, Fig. 9 As shown, a target detection method based on cross-attention multi-scale fusion is provided, which is executed by a computer device, and can be specifically executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In an embodiment of the present application, the method includes the following steps 101 to 102.

[0042] Step 101, obtaining a visible light image and an infrared image of a target to be detected.

[0043] Step 102, based on the visible light image and the infrared image, target detection is performed using a trained cross-attention multi-scale fusion detection network to determine the position and category of the target to be detected.

[0044] Among them, Fig.10 As shown, the cross-attention multi-scale fusion detection network includes a backbone network, a background perception module, a cross-attention fusion module and a detection head.

[0045] The backbone network is used to extract features of different scales of the visible light image and the infrared image.

[0046] The background perception module is used to predict the illumination and contrast of the visible light image and the infrared image. The background perception module uses the illumination condition and the target contrast as prior knowledge to guide the fusion of the visible light modality and the infrared modality.

[0047] The cross attention fusion module is used to cross-fuse the features extracted by the backbone network according to the illumination and contrast predicted by the background perception module, using differential attention and homogeneous attention to obtain a multimodal fusion feature map. The cross attention fusion module consists of two parts: differential attention and homogeneous attention. Differential attention enhances the difference between the visible light modality and the infrared modality, while homogeneous attention suppresses a large number of redundant features in the modality. The features of the two parts are then weightedly fused to obtain a multimodal fusion feature map.

[0048] The detection head is used to determine the position and category of the target to be detected according to the multimodal fusion feature map. The detection head generates a detection result by feature weighting.

[0049] This application uses a background-guided cross-attention multi-scale fusion detection network to achieve adaptive fusion of visible light and infrared features. First, the background perception module is used to calculate the illumination and contrast weights. Subsequently, the feature map extracted by the backbone network is input into the cross-attention fusion module composed of differential attention and homogeneous attention. Homogeneous attention focuses on shared features within the modality to avoid introducing a large number of redundant features; differential attention enhances the difference features between modalities to promote the fusion of complementary information, so as to enhance the inherent and discriminative features within and between modalities. Then, the enhanced features are summed according to the background perception weights to obtain a merged multimodal fusion feature map. Finally, the multimodal fusion feature map is input into the detection head to generate the detection result.

[0050] In an exemplary embodiment, step 102 includes the following steps 201 to 204 .

[0051] Step 201 , using a background perception module to perform illumination and contrast prediction on the visible light image and the infrared image, and determine a visible light illumination weight, an infrared illumination weight, a visible light contrast weight, and an infrared contrast weight.

[0052] The factors that affect the reliability of the cross-attention multi-scale fusion detection network are complex and diverse. Although illumination information is effective for cross-modal fusion, using only illumination information still has the following limitations:

[0053] 1) Methods based on lighting perception perform better in fixed scenes such as day and night, but it is difficult to distinguish targets and surrounding areas at dusk, dawn or high-similarity backgrounds;

[0054] 2) Lighting information is usually used to guide the fusion of inter-modal features, but it is insufficient for enhancing intra-modal features.

[0055] Therefore, this application uses background perception weighting instead of lighting perception weighting. Fig.11 As shown, the background perception module calculates illumination and target contrast as prediction weights, thereby guiding the adaptive fusion of intra-modal and inter-modal features more efficiently. Fig.11 The blue box in the figure represents the convolutional layer, which consists of a convolutional module, an activation layer, and a pooling layer. The connected gray rectangular boxes represent the fully connected layer, and the last gray box represents a mathematical operation; the orange rectangular box represents the convolutional layer.

[0056] In a specific example, step 201 includes the following steps 301 to 303 .

[0057] Step 301 , predicting the light intensity of the visible light image to determine the light intensity at night and the light intensity during the day.

[0058] The background perception module plays a role in understanding different lighting conditions and background environments, guiding the complementary learning and adaptive fusion of the two modalities in the feature extraction and enhancement stage. Due to the difference in the spectrum, visible light images are more dependent on external light sources, so they vary greatly from day to night. Infrared images are the opposite. Their imaging method is passive imaging, which more reflects the radiation difference between the target and the background. Generally speaking, in daytime scenes, visible light images have higher imaging quality and contain more color and texture information. In night scenes, infrared images are clearer and the outline of the target can be obtained.

[0059] Therefore, we first design a classification network to predict the illumination information and guide the fusion of inter-modal information. The output is t d , represents the daytime light intensity. In natural scenes, the daytime lighting conditions are usually better than those at night, which means that the assigned weight is higher. d The larger it is, the better the lighting conditions are.

[0060] Step 302: Determine the visible light weight and the infrared light weight using a gate function according to the nighttime light intensity and the daytime light intensity.

[0061] The illumination probability value can be used to represent the perceptual weight contributed by different modal images, but due to the binary nature of the light source, the calculated probability value is mostly close to 0 or 1. If the network directly multiplies the feature map of the branch with this weight, the modality with a lower probability value will be greatly suppressed, thus losing the meaning of fusion. In order to optimize the weight, this application designs a gate function to readjust the weights of the two modalities.

[0062] Specifically, the gate function is:

[0063]

[0064] Among them, W v is the visible light illumination weight, W i is the infrared illumination weight, W v and W i The weight used to guide the fusion of visible light and infrared modalities, W v +W i =1,t d is the daytime light intensity, t n is the night light intensity, t d +t n =1, during the day t n Approaching 0, at night t n Approaching 1.

[0065] When the probability of daytime is higher, the weight used for the visible light modality will be greater than 1 / 2. And because the weight of the infrared modality is no longer close to 0 at this time, the fusion of the two modalities can be more fully promoted, and vice versa. The present application adjusts the image to a size of 128×128 and inputs it into the background perception module. The visible light image is input into the illumination prediction network, which includes a convolutional layer and a fully connected layer. After the convolutional layer, an activation layer and 2×2 adaptive pooling are added to compress and extract illumination features. The fully connected layer is then input and calculated to convert the output of the illumination prediction network into the required weights. Among them, the convolutional layer consists of a convolutional module, an activation layer, and a pooling layer.

[0066] Step 303 , performing contrast prediction on the visible light image and the infrared image respectively according to the grayscale value of each pixel in the visible light image and the grayscale value of each pixel in the infrared image, so as to determine a visible light contrast weight and an infrared contrast weight.

[0067] It is not enough to use only illumination information to guide modality fusion. For example, the similarity of color and texture between the target and background in visible light images and the similarity of temperature between the target and background in infrared images will affect the accuracy of detection. Therefore, this application introduces contrast as prior knowledge based on illumination weight to guide the fusion of intra-modality features.

[0068] Generally speaking, the higher the similarity between the target and the background, the more difficult it is to distinguish. The contrast is measured by the grayscale difference between the target and the surrounding background. Therefore, the target area in the visible light image and the infrared image is first determined by sliding the image frame (when the grayscale difference between a certain image frame and other image frames is large and exceeds the set threshold 2, the area is the target area), and then the target area is divided into a 3×3 grid.

[0069] Define a variable m to represent the pixel mean of the rest of the grid area except the target. The specific calculation process is as follows:

[0070]

[0071] Among them, m j is the grayscale mean of the pixels in the jth grid, j represents the grid number, j = 1 to 8, N j is the total number of pixels in the jth grid, k represents the sequence number of the pixels in each grid, is the gray value of the kth pixel in the jth grid.

[0072] The variable L is introduced to represent the maximum grayscale value of the pixel in the target area, and the following formula is used to describe the contrast difference.

[0073]

[0074] Among them, c j is the contrast of the jth grid, L j is the maximum grayscale value of the pixel in the jth grid.

[0075] By calculating the contrast, the target and background can be better distinguished, and the target area in the image can be enhanced and the background area can be suppressed in the subsequent cross-attention fusion module.

[0076] Using the formula C = max (c i ) to determine the contrast weight, that is, to obtain the visible light contrast weight C v and infrared contrast weight C i .

[0077] Step 202 , using a backbone network to extract features of different scales of the visible light image and the infrared image, respectively, to obtain visible light feature maps of different scales and infrared feature maps of different scales.

[0078] In a specific example, in order to optimize the feature extraction capability of the network and improve the robustness of the network in complex scenes, the backbone network uses a dual-branch network composed of Darknet53 and CSP modules to extract features of different scales of the visible light image and the infrared image, wherein the CSP module is as follows: Fig.12 shown.

[0079] Fig.10 In the figure, the blue rectangle represents the visible light image branch, and the orange rectangle represents the infrared image branch. Specifically, the boxes in the backbone network represent a convolution module and an improved CSP module. The convolution module consists of Conv2d (kernel 3, step size 2, padding 1), batch normalization and activation function, and the improved CSP module consists of convolution, segmentation, convolution and splicing operations. The input image features are 640*640*3 at the beginning. After the backbone network is input into the cross fusion module, they become 80*80*256, 40*40*512, and 40*40*512 respectively. Then they are input into the subsequent network, and the detection results are generated through the iteration of the convolution network.

[0080] Step 203, using a cross-attention fusion module to cross-fuse visible light feature maps of different scales and infrared feature maps of different scales according to the visible light illumination weight, the infrared illumination weight, the visible light contrast weight and the infrared contrast weight, using differential attention and homogeneous attention to obtain fused features.

[0081] In an exemplary embodiment, Fig.13 and Fig.14 As shown, step 203 includes the following steps 401 to 406.

[0082] Step 401, for a visible light feature map and an infrared feature map at any scale, calculate the differential features and homogeneous features of the visible light feature map and the infrared feature map.

[0083] Due to the difference in imaging methods, images of different modalities contain both differential information and complementary information, and how to fuse the two modal information is the key to multispectral target detection. However, most of the existing methods based on dual-branch networks use simple fusion schemes, such as addition, multiplication and splicing between pixels of different modalities, which cannot fully utilize the inherent features between and within modalities. In addition, rough combinations and connections will also increase the difficulty of network learning and fitting, resulting in a decrease in detection performance. Inspired by the differential amplifier circuit, the differential mode signal and the common mode signal are amplified and suppressed respectively. The background perception guided cross-attention fusion module proposed in this application consists of two parts: differential attention and homogeneous attention. Given a visible light feature map M V and infrared characteristic map MI , the differential features between classes and the homogeneous features within classes can be expressed as:

[0084]

[0085] Among them, M D is the differential feature, M C Homogeneous characteristics.

[0086] The differential feature can be regarded as the difference between the two modes, and the complementary specific features between the modes can be enhanced by subtracting them. Conversely, the homogeneous component can be regarded as the sum of the two modes, and the consistency features within the modes can be enhanced by adding them. Based on this, the present application defines two new hybrid modes, outputting feature maps with richer information to achieve the final fusion.

[0087] The inspiration of inter-class differential attention comes from the differential signal in the differential circuit. Differential attention calculates the difference between visible light and infrared modalities to extract the differential features between modalities. Subsequently, the differential features are enhanced through channel attention.

[0088] Step 402, using global maximum average pooling operation and global maximum pooling operation, globally enhance the differential features to obtain a channel attention map.

[0089] The differential features are encoded into the global vector through global maximum average pooling and global maximum pooling, extracting the texture information of the target and integrating the global spatial information. The global vector can represent the difference in channel characteristics between modalities. Specifically, the differential features are obtained by global maximum average pooling to obtain the first global vector g 1 , the second global vector g is obtained through global maximum pooling 2 , and then through the shared convolution operation, the first feature map s is obtained. 1 and the second feature map s 2 , and then the first feature map is added to the second feature map to obtain the channel attention map.

[0090] Step 403: multiply the channel attention map with the visible light feature map and the infrared feature map respectively, and use the visible light illumination weight and the infrared illumination weight to perform weighted fusion to obtain a differential attention feature map.

[0091] Specifically, the channel attention map is multiplied with the input features of the two modalities to obtain the feature map of the key area, the importance of the visible light and infrared modalities is learned, and the enhanced feature map is weighted fused according to the illumination weight to obtain the differential attention feature map:

[0092]

[0093] Among them, M DAis the differential attention feature map, which represents the features processed by the inter-class difference module, W v is the visible light illumination weight, W i is the infrared illumination weight, M V is the visible light characteristic map, M I is the infrared characteristic image, V DA is the channel attention map.

[0094] This application adaptively learns the importance of different modalities across channels through differentiation, compression, excitation and weighted fusion, and its generalization is also enhanced. It is worth noting that the cross-attention fusion module draws on the residual network to enhance the stability of the network. The feature map after differentiation is input into a specific modality through a jump connection. Through residual mapping, inter-class differential attention can avoid directly affecting the characteristics of a single modality while supplementing the complementary information between modalities.

[0095] Step 404: perform global average pooling and full connection operations on the homogeneous features in sequence to obtain a visible light modal attention map. and infrared modality attention map

[0096] In multispectral target detection tasks, the similarity between foreground and background also affects the detection performance. In addition to the complementary features between modalities, the inherent features within the modality are also crucial for the extraction of target discriminative features. Therefore, this application learns the inherent features within the modality under the guidance of contrast weights.

[0097] Step 405: multiply the visible light modal attention map with the visible light feature map, multiply the infrared modal attention map with the infrared feature map, and use the visible light contrast weight and the infrared contrast weight to perform weighted fusion to obtain a homogeneous attention feature map.

[0098] Specifically, the following formula is used to determine the homogeneous attention feature map:

[0099]

[0100] Among them, M CA is the homogeneous attention feature map, C v is the visible light contrast weight, C i is the infrared contrast weight, is the visible light modality attention map, is the infrared modality attention map.

[0101] This application achieves adaptive channel screening through summation, weight sharing, compression and normalization. In steps 404 and 405, the sharing of fully connected layer parameters is used to reduce the dimension of features and improve computational efficiency; through channel screening and guidance of prior knowledge, the weights of the model are redistributed to the feature channels of visible light and infrared modalities to avoid introducing a large number of redundant features; the use of jump connections enables the network to reuse shallow features and improves the ability to represent complex features.

[0102] Step 406, the differential attention feature maps of different scales and the homogeneous attention feature maps of different scales are fused to obtain a multimodal fusion feature map.

[0103] Generally speaking, a dual-branch network can choose to process features in serial or parallel. However, a large number of studies have shown that simple serial-parallel methods will aggravate the imbalance of the network and thus affect the performance of the network. In addition, changes in target scale, especially some small-sized targets, will also lead to a decline in performance. Therefore, this application designs a multi-scale fusion strategy, which realizes the cross-modal fusion detection of visible light and infrared images through the intersection of feature maps and the fusion of three different scales of small, medium and large feature maps. In the feature extraction stage of the network, feature maps from different modalities and scales (small, medium and large) are extracted and sent to the cross-attention fusion module, followed by feature weighting to achieve multi-scale feature fusion. The subsequent fusion steps are shown in the following formula:

[0104]

[0105] Among them, F FUSE is a multimodal fusion feature map, P is the number of scales, where P = 3, representing small, medium, and large scales. is the homogeneous attention feature map of the p-th scale, is the differential attention feature map of the p-th scale.

[0106] Specifically, the feature maps of the two modal branch networks are input into the cross-attention fusion module through horizontal connections. To maintain the weight of the global feature, a convolutional layer is used after each result to reduce its dimension to 1 / 2 of the original, and then the feature map is bilinearly interpolated to restore it to the length and width of the input. Finally, the features of different scales are spliced ​​together as global features of multi-scale fusion. It is worth noting that the features of both visible light and infrared modalities have been processed and refined by the cross-attention fusion module to avoid the loss of key information in the process of feature extraction and attention map generation.

[0107] Step 204: According to the multimodal fusion feature map, a detection head is used to determine the position and category of the target to be detected.

[0108] In an exemplary embodiment, in order to enable the cross-attention multi-scale fusion detection network to adaptively capture inter-modal and intra-modal information according to background conditions, a background perception loss is introduced. It is worth noting that contrast, as a priori knowledge, does not require independent network training, so no supervision is required and it does not need to be part of the loss function. Background perception loss includes detection loss and illumination condition loss. Detection loss is used to calibrate multispectral detection results, and illumination condition loss is used to supervise inter-modal differential weights.

[0109] The exact definition of background perception loss is as follows:

[0110] Loss = L d +L l ;

[0111] Among them, Loss is the background perception loss value, L d is the detection loss value, L l Loss of light conditions.

[0112] Detection loss includes classification loss, bounding box regression loss and confidence loss.

[0113] The classification loss uses the Binary Cross Entropy Loss (BCE) function. For the objects in the image, the strong association of the states enables the labels to better guide the network to learn the classification capabilities. The classification loss is defined as: L cls =-[y a log(p a )+(1-y a )log(1-p a )].

[0114] The border regression loss uses the Generalized Intersection over Union (GIoU) ​​loss. In order to alleviate the gradient problem of IoU loss when the detection boxes do not overlap, GIoU adds a penalty term on the basis of the original IoU loss, which is defined as:

[0115] The confidence loss is based on FocalLoss, which mainly solves the problem of severe imbalance in the proportion of different targets in the data set. It is suitable for complex scenarios such as small samples and large target scales. It is defined as: L conf =-α a (1-p a ) τ log(p a ).

[0116] Among them, a represents the serial number of the sample, p a Represents the probability predicted by the model, ya It is a binary variable with a value of 0 or 1. A represents the area of ​​the real box, B represents the area of ​​the predicted box, and D represents the area of ​​the smallest box that contains both. a It mainly solves the imbalance of positive and negative samples, and τ mainly solves the imbalance of difficult and easy samples. Here α a =0.25,τ=2.

[0117] The fusion of inter-modal complementarity and intrinsic information within the modality depends largely on the guidance of the background perception module, especially the perception of lighting conditions. The lighting condition reflects the strength of the lighting conditions in the image, and can be regarded as a classifier to calculate the probability of belonging to daytime and nighttime. Therefore, the cross entropy loss is used to constrain its training process, which means: L l =-zlogσ(x)-(1-z)log(1-σ(x)).

[0118] Among them, z is the lighting condition label of the input image, x represents the probability that the image belongs to daytime, and σ is the softmax function, which mainly normalizes the lighting condition probability to [0,1]. In order to more fully characterize the strength of the lighting conditions, this application sets the value space of z to 0, 0.5, and 1.0, representing night, low light, and daytime scenes, respectively.

[0119] The target detection method based on cross-attention multi-scale fusion provided by this application can be inherited on a multispectral detector. In addition, this application has conducted a large number of comparative experiments on three multispectral detection datasets (LLVIP, FLIR and VEDAI), and the results show that the multispectral detector provided by this application has achieved higher detection accuracy than the most advanced multispectral detector currently available.

[0120] The following first introduces the dataset and evaluation indicators; then introduces the experimental settings and algorithm deployment; then, compares the cross-attention multi-scale fusion-based object detection method provided by this application with other advanced methods in terms of mAP and mAP on three datasets. 50 A large number of comparative experiments were carried out on indicators such as , and finally an ablation experiment was conducted.

[0121] (1) Dataset.

[0122] This application compares the public dual-light fusion detection data sets, focusing on the analysis of three data sets with remote sensing, road, and low-light backgrounds, namely FLIR, LLVIP, and VEDAI. Fig.15 Part (a) shows the distribution and scale variation of the width of the VEDAI dataset. Fig.15 Part (b) shows the distribution and scale variation of the width of the FLIR dataset. Fig.15Part (c) shows the distribution and scale variation of the width of the LLVIP dataset. Fig.16 Part (a) shows the height distribution and scale variation of the VEDAI dataset. Fig.16 Part (b) shows the distribution and scale variation of the height of the FLIR dataset. Fig.16 Part (c) shows the height distribution and scale variation of the LLVIP dataset, indicating that the dataset used in this application covers instances of different scales and is quite representative.

[0123] The VEDAI dataset is mainly used for target detection in optical remote sensing images. The dataset is shot in a variety of complex backgrounds such as cities, roads, fields, and forests, covering 9 different types of small and medium-sized vehicles, with a total of 1,246 pairs of images and 3,640 instances. This application is carried out on images with a resolution of 512*512, which is characterized by large differences in target size, and can verify the algorithm's ability to integrate detection of targets of different scales.

[0124] FLIR mainly has three types of targets: pedestrians, bicycles, and cars, and their backgrounds are mainly on typical streets and highways. The FLIR aligned dataset manually deleted the unaligned images based on the original dataset to avoid the problem of network training failing to converge. This application is all performed on the FLIR aligned dataset. For convenience, subsequent FLIR refers to the aligned version. The FLIR aligned dataset contains 5142 pairs of bimodal images, and is divided into training, validation, and test datasets in a ratio of 6:2:2. The sampling frame rate is mostly 2 frames per second, which is mainly used in autonomous driving scenarios.

[0125] The LLVIP dataset is designed to solve the problem of multispectral pedestrian detection in low-light scenes. The ratio of night and day scenes in the dataset reaches 12:1. The dataset contains 15,488 image pairs, of which 9,292 pairs are used for training and 3,098 pairs are used for verification and testing respectively. Compared with other datasets, the images of visible light and infrared modalities in the LLVIP dataset are strictly aligned in time and space, which allows the network to focus more on improving the fusion detection performance without considering the errors caused by calibration.

[0126] (2) Experimental results

[0127] In order to verify the effectiveness of the cross-attention multi-scale fusion detection network provided by this application, it is compared with baseline detectors and state-of-the-art multispectral detection networks, including GAFF, CFT and other methods. The baseline detectors include one-stage detection algorithms YOLOv9 and Faster R-CNN and two-stream networks that only use pixel addition for fusion.

[0128] 1) VEDAI dataset: In the remote sensing scenario, the method of this application is compared with other related works, and the experimental results are shown in Table 1. It can be observed that on the VEDAI dataset, the method of this application achieves the best detection performance compared with other algorithms. For single-modal detection, the method of this application improves the mAP by 15.2% and 18.2% respectively over the first-stage YOLOv9 and Faster R-CNN. For multi-modal detection, the method of this application is 10.2% and 11.3% higher in mAP. 50 The mAP and mAP indicators are 1.3% and 0.7% higher than the best method respectively.

[0129] Table 1 Comparison of experimental results on the VEDAI dataset

[0130]

[0131] In addition to quantitative comparison, this application also performs qualitative analysis on the VEDAI dataset. Fig.17 and Fig.18 The original input image, the baseline, and the detection results of this application are shown. The true value is marked in the input infrared image. Fig.17 Part (a) and Fig.18 Part (a) of is the original input visible light image, as shown in Fig.17 Part (b) and Fig.18 Part (b) of is the original input infrared image, as shown in Fig.17 Part (c) and Fig.18 Part (c) of the diagram shows the detection result of the baseline. Fig.17 Part (d) and Fig.18 Part (d) shows a schematic diagram of the detection results of the present application. In remote sensing scenes, the target size is small, the arrangement is dense, and the texture features are weak. Using only a simple two-stream network can easily lead to false detection or missed detection (the area where the triangle is located). However, the method of the present application fully integrates the complementary and inherent characteristics of the two modes, which can significantly reduce the occurrence of false detection and missed detection. Therefore, the present application has better detection performance in remote sensing scenes.

[0132] 2) FLIR dataset: In the road scene, the results of the method of this application are also compared with those of other methods, as shown in Table 2. It can be seen that the performance of this application is the best in the FLIR-aligned dataset. In particular, mAP 50 In terms of indicators, this application is 2.2% higher than the typical two-stream network LRAF-Net. In addition, compared with the single-modal detection algorithm, this application is 2.2% higher in mAP. 50 The mAP and mAP indicators are 8.6% and 4.0% higher respectively. This shows that the fusion algorithm of this application is effective and greatly improves the performance of single-modal detection.

[0133] Table 2 Comparison of experimental results of FLIR dataset

[0134]

[0135] Similarly, this application conducts a qualitative analysis on the FLIR dataset and selects two scenes, daytime and nighttime. The detection results are as follows: Fig.19 and Fig. 20 As shown in Figure 1. The true value of the target is marked by the white rectangular box in the infrared image. The targets in the road scene occlude each other, which leads to the degradation of the algorithm performance. Fig.19 Part (a) of is the visible light image input for the daytime scene, such as Fig.19 Part (b) of is the infrared image input for the daytime scene, such as Fig.19 Part (c) of the figure is a schematic diagram of the detection results of the baseline in the daytime scene. Fig.19 Part (d) of FIG. 1 is a schematic diagram of the detection results of the present application in a daytime scene, such as Fig. 20 Part (a) of is the visible light image input for the night scene, such as Fig. 20 Part (b) of is the infrared image input of the night scene, such as Fig. 20 Part (c) of FIG. 1 is a schematic diagram of the detection result of the baseline of the night scene, as shown in FIG. Fig. 20 Part (d) is a schematic diagram of the detection results of the present application in the night scene. In the baseline detection results, pedestrians overlapping with electric poles in the daytime scene were not detected, and vehicles blocked by pedestrians in the night scene were detected twice. In the detection results of the present application, all the above targets were detected. The reason is that the method of the present application takes into account the global and local information of the target, and effectively enhances the characteristics of the target under occlusion.

[0136] 3) LLVIP dataset: In low-light scenarios, the comparison results of this application and other algorithms are shown in Table 3. It can be seen that compared with other algorithms, this application has the most advanced performance on the LLVIP dataset, with mAP 50 The mAP reaches 97.9% and 69.2%. In particular, the mAP index is 7.3% and 2.9% higher than that of YOLOv9 and LRAF-Net, respectively. In addition, from the single-modal detection results, the mAP of infrared images is significantly higher than that of visible light images, which shows that infrared images have better feature expression capabilities in low-light conditions.

[0137] Table 3 Comparison of experimental results on the LLVIP dataset

[0138]

[0139]

[0140] Fig.21 and Fig. 22 The following table shows some experimental results of the baseline and this application. It can be seen that infrared images have better detection performance in low-light scenes and can intuitively show the difference between foreground and background. Fig.21 Part (a) and Fig. 22 Part (a) of is the original input visible light image, as shown in Fig.21 Part (b) and Fig. 22 Part (b) of is the original input infrared image, as shown in Fig.21 Part (c) and Fig. 22 Part (c) of the diagram shows the detection result of the baseline. Fig.21 Part (d) and Fig. 22 Part (d) of FIG. 1 is a schematic diagram of the detection results of the present application. When the targets overlap or are blocked, especially in Fig.21 Part (c) and Fig. 22 In part (c), there is light source interference next to the target, and the baseline is likely to miss detection. This application effectively solves the above problem by deeply fusing the information of the two modalities instead of simply adding the two modalities like the baseline.

[0141] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0142] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0143] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0144] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0145] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ReadOnlyMemory, ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (Magnetoresistive RandomAccess Memory, MRAM), ferroelectric random access memory (Ferroelectric RandomAccess Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (RandomAccess Memory, RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0146] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0147] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A target detection method based on cross-attention multi-scale fusion, characterized in that: The target detection method based on cross-attention multi-scale fusion includes: Acquire visible light images and infrared images of the target to be detected; According to the visible light image and the infrared image, target detection is performed using a trained cross-attention multi-scale fusion detection network to determine the position and category of the target to be detected; Among them, the cross-attention multi-scale fusion detection network includes a backbone network, a background perception module, a cross-attention fusion module and a detection head; the backbone network is used to extract features of different scales of the visible light image and the infrared image; the background perception module is used to predict the illumination and contrast of the visible light image and the infrared image; the cross-attention fusion module is used to cross-fuse the features extracted by the backbone network according to the illumination and contrast predicted by the background perception module, using differential attention and homogeneous attention to obtain a multimodal fusion feature map; the detection head is used to determine the position and category of the target to be detected according to the multimodal fusion feature map.

2. The target detection method based on cross-attention multi-scale fusion according to claim 1 is characterized in that: According to the visible light image and the infrared image, target detection is performed using a trained cross-attention multi-scale fusion detection network to determine the position and category of the target to be detected, specifically including: Using a background perception module to predict illumination and contrast of the visible light image and the infrared image, and determine a visible light illumination weight, an infrared light weight, a visible light contrast weight, and an infrared contrast weight; Using a backbone network to extract features of different scales of the visible light image and the infrared image, respectively, to obtain visible light feature maps of different scales and infrared feature maps of different scales; A cross-attention fusion module is used to cross-fuse visible light feature maps of different scales and infrared feature maps of different scales according to the visible light illumination weight, the infrared illumination weight, the visible light contrast weight and the infrared contrast weight, using differential attention and homogeneous attention to obtain a multimodal fusion feature map; According to the multimodal fusion feature map, a detection head is used to determine the position and category of the target to be detected.

3. The target detection method based on cross-attention multi-scale fusion according to claim 2 is characterized in that: The background perception module is used to predict the illumination and contrast of the visible light image and the infrared image, and the visible light illumination weight, infrared light weight, visible light contrast weight and infrared contrast weight are determined, specifically including: Predicting the light intensity of the visible light image to determine the light intensity at night and the light intensity during the day; Determine the visible light weight and the infrared light weight using a gate function according to the nighttime light intensity and the daytime light intensity; According to the grayscale value of each pixel in the visible light image and the grayscale value of each pixel in the infrared image, contrast prediction is performed on the visible light image and the infrared image respectively to determine a visible light contrast weight and an infrared contrast weight.

4. The target detection method based on cross-attention multi-scale fusion according to claim 3 is characterized in that: The gate function is: Among them, W v is the visible light illumination weight, W i is the infrared illumination weight, t d is the daytime light intensity, t n is the night light intensity.

5. The target detection method based on cross-attention multi-scale fusion according to claim 2 is characterized in that: The backbone network uses a dual-branch network composed of Darknet53 and CSP structures to extract features of different scales of the visible light image and the infrared image.

6. The target detection method based on cross-attention multi-scale fusion according to claim 2 is characterized in that: A cross-attention fusion module is used to cross-fuse visible light feature maps of different scales and infrared feature maps of different scales according to the visible light illumination weight, the infrared illumination weight, the visible light contrast weight and the infrared contrast weight, using differential attention and homogeneous attention to obtain a multimodal fusion feature map, specifically including: For a visible light feature map and an infrared feature map at any scale, calculating differential features and homogeneous features between the visible light feature map and the infrared feature map; Using a global maximum average pooling operation and a global maximum pooling operation, the differential features are globally enhanced to obtain a channel attention map; The channel attention map is multiplied by the visible light feature map and the infrared feature map respectively, and the visible light illumination weight and the infrared illumination weight are used for weighted fusion to obtain a differential attention feature map; Performing global average pooling and full connection operations on the homogeneous features in sequence to obtain a visible light modal attention map and an infrared modal attention map; Multiplying the visible light modal attention map with the visible light feature map, multiplying the infrared modal attention map with the infrared feature map, and performing weighted fusion using the visible light contrast weight and the infrared contrast weight to obtain a homogeneous attention feature map; The differential attention feature maps of different scales and the homogeneous attention feature maps of different scales are fused to obtain a multimodal fusion feature map.

7. The target detection method based on cross-attention multi-scale fusion according to claim 6 is characterized in that: The differential attention feature map is determined using the following formula: The homogeneous attention feature map is determined using the following formula: Among them, M DA is the differential attention feature map, M CA is the homogeneous attention feature map, W v is the visible light illumination weight, W i is the infrared illumination weight, M V is the visible light characteristic map, M I is the infrared characteristic image, V DA is the channel attention map, C v is the visible light contrast weight, C i is the infrared contrast weight, is the visible light modality attention map, is the infrared modality attention map.

8. The target detection method based on cross-attention multi-scale fusion according to claim 1 is characterized in that: The loss function during the training of the cross-attention multi-scale fusion detection network includes detection loss and illumination condition loss; the detection loss includes classification loss, border regression loss and confidence loss.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the target detection method based on cross-attention multi-scale fusion as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the target detection method based on cross-attention multi-scale fusion described in any one of claims 1 to 8 is implemented.

Images