Visible light and infrared image target detection method and device based on dual-stream attention

Abstract

The present application discloses a method and device for target detection in visible light and infrared images based on dual-stream attention, which relates to the technical field of target detection in visible light and infrared images. The visible light and infrared image data are aligned, and a 3×3 convolutional layer is used to preliminarily extract a shallow feature map. The shallow feature map is then input into a residual network composed of a dual-stream Transformer module to extract a deep feature map. The deep feature map, the shallow feature map and the original image are then aggregated, and a detection head is used to achieve target detection. The present application considers both channel and spatial attention, so that the target detection network model can not only simultaneously enhance the representation capabilities of target features in the channel dimension and the spatial dimension, improve the accuracy and robustness of target detection, but also better adapt to targets of different sizes and improve the generalization capability of target detection. Moreover, the present application comprehensively utilizes the advantages of visible light and infrared images to accurately extract features and achieve accurate detection in complex backgrounds, low contrast or occlusion conditions.

Description

Technical Field

[0001] The present application relates to the technical field of visible light and infrared image target detection, and in particular to a visible light and infrared image target detection method and device based on dual-stream attention. Background Art

[0002] With the rapid development of computer networks, object detection, as the cornerstone of image understanding and computer vision, is the basis for solving more complex and higher-level visual tasks such as segmentation, scene understanding, object tracking, image description, event detection, and activity recognition. Object detection has a wide range of applications in many fields of artificial intelligence and information technology, including robot vision, consumer electronics, security, autonomous driving, human-computer interaction, content-based image retrieval, intelligent video surveillance, and augmented reality.

[0003] In the related art, target detection is usually achieved by using traditional target detection algorithms that rely on manually extracted features, target detection methods based on CNN (Convolutional Neural Networks), target detection methods based on the original Transformer (sequence model based on attention mechanism), and target detection methods based on single infrared or visible light. Among them, traditional target detection algorithms usually require manual design of feature extractors, such as Haar features, HOG (Histogram of Oriented Gradients) features, and then use machine learning algorithms to train classifiers. However, the applicant recognizes that traditional target detection algorithms require a lot of manual work and have limitations for complex scenes and ever-changing targets; existing target detection network models have problems with image detail information loss and image edge extraction fuzziness, resulting in poor generalization ability and detection effect of target detection; and in complex backgrounds, low contrast or occlusion, the use of single infrared or visible light target detection methods cannot accurately extract features, which also leads to low generalization ability and accuracy of target detection. Summary of the invention

[0004] In view of this, the present application provides a visible light and infrared image target detection method and device based on dual-stream attention, the main purpose of which is to solve the following problems: traditional target detection algorithms require a lot of manual work and have limitations for complex scenes and ever-changing targets; existing target detection network models have problems with image detail information loss and fuzzy image edge extraction, resulting in poor generalization ability and detection effect of target detection; and in complex backgrounds, low contrast or occlusion conditions, the use of single infrared or visible light target detection methods cannot accurately extract features, which also leads to the problem of low generalization ability and accuracy of target detection.

[0005] According to the first aspect of the present application, a method for target detection in visible light and infrared images based on dual-stream attention is provided, the method comprising:

[0006] In response to a target detection request, obtaining a visible light image and an infrared image carried in the target detection request, performing a data alignment operation on the visible light image and the infrared image to obtain a visible light input image and an infrared input image;

[0007] A shallow feature extraction module based on the target detection network model performs preliminary feature extraction on the visible light input map and the infrared input map to obtain a shallow visible light feature map and an infrared shallow feature map; a deep feature extraction module based on the target detection network model performs deep feature extraction on the shallow visible light feature map and the infrared shallow feature map to obtain a deep visible light feature map and an infrared deep feature map, wherein the deep feature extraction module includes a channel attention unit, a spatial attention unit, and a dual-stream feedforward neural network unit;

[0008] A feature fusion module based on the target detection network model aggregates the visible light input map, the infrared input map, the visible light shallow feature map, the infrared shallow feature map, the visible light deep feature map, and the infrared deep feature map to obtain a visible light feature map and an infrared feature map;

[0009] The detection head based on the target detection network model predicts the visible light feature map and the infrared feature map to obtain a target detection result, and pushes the target detection result to the user who initiated the target detection request.

[0010] According to a second aspect of the present application, a device for detecting targets in visible light and infrared images based on dual-stream attention is provided, the device comprising:

[0011] a preprocessing module, configured to, in response to a target detection request, obtain a visible light image and an infrared image carried in the target detection request, perform a data alignment operation on the visible light image and the infrared image, and obtain a visible light input image and an infrared input image;

[0012] A feature extraction module, used to perform preliminary feature extraction on the visible light input map and the infrared input map based on the shallow feature extraction module of the target detection network model to obtain a visible light shallow feature map and an infrared shallow feature map, and perform deep feature extraction on the visible light shallow feature map and the infrared shallow feature map based on the deep feature extraction module of the target detection network model to obtain a visible light deep feature map and an infrared deep feature map, wherein the deep feature extraction module includes a channel attention unit, a spatial attention unit, and a dual-stream feedforward neural network unit;

[0013] A feature aggregation module, configured to aggregate the visible light input map, the infrared input map, the visible light shallow feature map, the infrared shallow feature map, the visible light deep feature map, and the infrared deep feature map based on the feature fusion module of the target detection network model to obtain a visible light feature map and an infrared feature map;

[0014] The feature detection module is used to predict the visible light feature map and the infrared feature map based on the detection head of the target detection network model, obtain the target detection result, and push the target detection result to the user who initiated the target detection request.

[0015] By means of the above technical scheme, the present application provides a method and device for target detection in visible light and infrared images based on dual-stream attention. The present application responds to a target detection request, obtains a visible light image and an infrared image carried in the target detection request, performs data alignment operations on the visible light image and the infrared image, obtains a visible light input map and an infrared input map, performs preliminary feature extraction on the visible light input map and the infrared input map based on a shallow feature extraction module of a target detection network model, obtains a visible light shallow feature map and an infrared shallow feature map, and performs deep feature extraction on the visible light shallow feature map and the infrared shallow feature map based on a deep feature extraction module of a target detection network model. The deep feature extraction module obtains the deep feature map of visible light and the deep feature map of infrared. The feature fusion module based on the target detection network model aggregates the visible light input map, the infrared input map, the visible light shallow feature map, the infrared shallow feature map, the visible light deep feature map, and the infrared deep feature map to obtain the visible light feature map and the infrared feature map. The detection head based on the target detection network model predicts the visible light feature map and the infrared feature map to obtain the target detection result, and pushes the target detection result to the user who initiated the target detection request, wherein the deep feature extraction module includes a channel attention unit, a spatial attention unit, and a dual-stream feedforward neural network unit. The present application performs data alignment on the input visible light and infrared images, and then uses a 3×3 convolutional layer to preliminarily extract the shallow feature map from the processed data, and inputs the obtained shallow feature map into the residual network composed of a dual-stream Transformer module to extract the deep feature map, and then aggregates the deep feature map, the shallow feature map and the original image, and inputs the aggregated feature map into the detection head to achieve target detection. This application considers both channel attention and spatial attention, so that the target detection network model can not only enhance the representation ability of target features in the channel dimension and spatial dimension at the same time, thereby improving the accuracy and robustness of target detection, but also better adapt to targets of different sizes and improve the generalization ability of target detection. Moreover, this application comprehensively utilizes the advantages of visible light images and infrared images, can accurately extract features, improve the generalization ability and robustness of the target detection network model, and thus achieve accurate target detection in complex backgrounds, low contrast or occlusion.

[0016] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0018] Figure 1 A schematic flow chart of a method for target detection in visible light and infrared images based on dual-stream attention provided in an embodiment of the present application is shown;

[0019] Figure 2A A schematic diagram of another method flow for target detection in visible light and infrared images based on dual-stream attention provided in an embodiment of the present application is shown;

[0020] Figure 2B A schematic diagram of a dual-stream Transformer module model provided in an embodiment of the present application is shown;

[0021] Figure 2C A schematic diagram of a channel attention unit model provided in an embodiment of the present application is shown;

[0022] Figure 2D A schematic diagram of a spatial attention unit model provided in an embodiment of the present application is shown;

[0023] Figure 2E A schematic diagram of a dual-stream feedforward neural network unit model provided in an embodiment of the present application is shown;

[0024] Figure 2F A schematic diagram of a network flow for visible light and infrared image target detection based on dual-stream attention provided in an embodiment of the present application is shown;

[0025] Figure 2G A schematic diagram of another network model for visible light and infrared image target detection based on dual-stream attention provided in an embodiment of the present application is shown;

[0026] Figure 3A A schematic diagram of the structure of a visible light and infrared image target detection based on dual-stream attention provided in an embodiment of the present application is shown;

[0027] Figure 3BA schematic diagram of another structure of visible light and infrared image target detection based on dual-stream attention provided in an embodiment of the present application is shown;

[0028] Figure 4 A schematic diagram of the device structure of a computer device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0029] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0030] The present application embodiment provides a method for detecting targets in visible light and infrared images based on dual-stream attention. Figure 1 As shown, the method includes:

[0031] 101. In response to a target detection request, obtain a visible light image and an infrared image carried in the target detection request, perform data alignment operations on the visible light image and the infrared image, and obtain a visible light input image and an infrared input image.

[0032] In the field of object detection, traditional object detection methods usually require manually designing feature extractors, such as Haar features, HOG features, etc., and then using machine learning algorithms to train classifiers. This method requires a lot of manual work and has limitations for complex scenes and changing targets. Deep learning can automatically learn features in images by building deep structures such as convolutional neural networks, and can extract and combine features layer by layer to achieve more accurate and robust object detection. However, some current classic deep learning object detection algorithms, such as Faster R-CNN (a deep neural network architecture for object detection), YOLO (You Only Look Once, an object recognition and positioning algorithm based on a deep neural network), SSD (Single Shot MultiBox Detector, a method for detecting objects in images using a single deep neural network), etc., still have problems with image detail information loss and edge extraction fuzziness, which will ultimately affect the image detection effect, and the generalization of the algorithm needs to be further improved.

[0033] To solve this problem, this application proposes a visible light and infrared image target detection method based on dual-stream attention, which combines the dual-stream attention of space and channels, and improves the detection effect of visible light and infrared images by combining visible light and infrared dual-task training. It can solve the limitations of traditional methods and CNN-based methods in the field of target detection, and use Transformer to expand the network's perception domain, which can parse global information and effectively enhance the network's representation ability. In addition, the introduction of dual-stream attention can not only explore the local and non-local similarities of space, but also consider the connection between space and channels to achieve collaborative enhancement of features. The improved two-stream feedforward neural network can effectively fuse original, shallow, and deep information, retain more details, and thus improve detection accuracy. The executor of this application may be a target detection system, which relies on the computing power of the server to provide services to users. The server may be an independent server or a server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, so as to accurately and quickly detect targets in images.

[0034] In an embodiment of the present application, the target detection system responds to the target detection request and obtains the visible light image and infrared image carried by the target detection request. Due to the difference in spatial and spectral resolution between visible light and infrared images, the target detection system performs data alignment operations on the visible light image and the infrared image to obtain a visible light input image and an infrared input image. The use of visible light and infrared dual-source data can provide a variety of feature expressions in target detection tasks. By fusing the features of the two, the target detection system can enhance performance and improve the generalization ability and robustness of the model.

[0035] 102. A shallow feature extraction module based on the target detection network model performs preliminary feature extraction on the visible light input map and the infrared input map to obtain a visible light shallow feature map and an infrared shallow feature map. A deep feature extraction module based on the target detection network model performs deep feature extraction on the visible light shallow feature map and the infrared shallow feature map to obtain a visible light deep feature map and an infrared deep feature map.

[0036] In an embodiment of the present application, the target detection system performs preliminary feature extraction on the visible light input map and the infrared input map based on the shallow feature extraction module of the target detection network model to obtain the visible light shallow feature map and the infrared shallow feature map. Then, the target detection system performs deep feature extraction on the visible light shallow feature map and the infrared shallow feature map based on the deep feature extraction module of the target detection network model to obtain the visible light deep feature map and the infrared deep feature map. Among them, the deep feature extraction module is composed of a residual Transformer layer, including a channel attention unit, a spatial attention unit, and a dual-stream feedforward neural network unit. The use of Transformer can expand the network's perception domain, parse global information, and effectively enhance the network's representation ability. Moreover, the introduction of dual-stream attention, namely channel attention and spatial attention, in the deep feature extraction module can not only explore the local and non-local similarities of space, but also take into account the connection between space and channels, and realize collaborative enhancement of features, thereby improving the accuracy and robustness of target detection.

[0037] 103. The feature fusion module based on the target detection network model aggregates the visible light input map, the infrared input map, the visible light shallow feature map, the infrared shallow feature map, the visible light deep feature map, and the infrared deep feature map to obtain the visible light feature map and the infrared feature map.

[0038] In an embodiment of the present application, the target detection system aggregates the visible light input map, the infrared input map, the visible light shallow feature map, the infrared shallow feature map, the visible light deep feature map, and the infrared deep feature map based on the feature fusion module of the target detection network model to obtain a visible light feature map and an infrared feature map. By effectively fusing the original, shallow, and deep information, the target detection system can retain more detailed features, achieve high-quality information fusion, and thus improve detection accuracy.

[0039] 104. The detection head based on the target detection network model predicts the visible light feature map and the infrared feature map to obtain the target detection result, and pushes the target detection result to the user who initiates the target detection request.

[0040] In the embodiment of the present application, the target detection system predicts the visible light feature map and the infrared feature map based on the detection head of the target detection network model, obtains the target detection result, and pushes the target detection result to the user who initiated the target detection request. Among them, the present application adopts a multi-layer perceptron as the detection head of the target detection network model, based on which accurate and robust detection effects of dual tasks can be achieved.

[0041] The method provided in the embodiment of the present application, in response to a target detection request, obtains a visible light image and an infrared image carried in the target detection request, performs data alignment operations on the visible light image and the infrared image to obtain a visible light input map and an infrared input map, performs preliminary feature extraction on the visible light input map and the infrared input map based on a shallow feature extraction module of a target detection network model to obtain a visible light shallow feature map and an infrared shallow feature map, performs deep feature extraction on the visible light shallow feature map and the infrared shallow feature map based on a deep feature extraction module of a target detection network model to obtain a visible light deep feature map and an infrared deep feature map, a feature fusion module based on the target detection network model aggregates the visible light input map, the infrared input map, the visible light shallow feature map, the infrared shallow feature map, the visible light deep feature map, and the infrared deep feature map to obtain a visible light feature map and an infrared feature map, a detection head based on the target detection network model predicts the visible light feature map and the infrared feature map to obtain a target detection result, and pushes the target detection result to a user who initiates the target detection request, wherein the deep feature extraction module includes a channel attention unit, a spatial attention unit, and a dual-stream feedforward neural network unit. This application performs data alignment of the input visible light and infrared images, and then uses a 3×3 convolutional layer to preliminarily extract shallow feature maps from the processed data, and inputs the obtained shallow feature maps into a residual network composed of a two-stream Transformer module to extract deep feature maps, and then aggregates the deep feature maps, shallow feature maps, and original images, and inputs the aggregated feature maps into the detection head to achieve target detection. This application considers both channel attention and spatial attention, so that the target detection network model can not only enhance the representation capabilities of target features in the channel dimension and spatial dimension at the same time, thereby improving the accuracy and robustness of target detection, but also better adapt to targets of different sizes and improve the generalization ability of target detection. Moreover, this application comprehensively utilizes the advantages of visible light images and infrared images, can accurately extract features, improve the generalization ability and robustness of the target detection network model, and thus achieve accurate target detection in complex backgrounds, low contrast or occlusion conditions.

[0042] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, the embodiment of the present application provides another visible light and infrared image target detection method based on dual-stream attention, such as Figure 2A As shown, the method includes:

[0043] 201. In response to a target detection request, obtain a visible light image and an infrared image carried in the target detection request, perform data alignment operations on the visible light image and the infrared image, and obtain a visible light input image and an infrared input image.

[0044] In the embodiment of the present application, the size of the visible light image is 1920×1080×3, and the size of the infrared image is 640×512×1. Due to the difference in spatial and spectral resolutions of visible light and infrared images, it is necessary to segment the visible light image and the infrared image so that their resolutions are the same, so that the target detection system can accurately extract features. Specifically, in the data alignment stage, the target detection system obtains a preset spatial resolution, segments the visible light image according to the preset spatial resolution, and obtains a visible light input image. The infrared image is segmented according to the preset spatial resolution to obtain a segmented infrared image. Among them, the preset spatial resolution is 64×64, and the target detection system can segment the visible light and infrared images into sub-images with overlapping areas with a spatial resolution of 64×64. At this time, the size of the visible light image is 64×64×3, and the size of the infrared image is 64×64×1. It should be noted that because there are target objects in both visible light and infrared images, the target objects are in the sub-images with overlapping areas, which can filter out invalid images, narrow the detection range, and improve the accuracy of target detection. Next, since the number of channels of the segmented infrared image and the visible light image is different, the target detection system uses a channel interpolation algorithm to expand the segmented infrared image according to the preset number of channels to obtain an infrared input image. The preset number of channels is 3, and the target detection system can use the channel interpolation algorithm to expand the number of channels of the infrared image to 3, and finally obtain a visible light input image and an infrared input image. After the data alignment process, the target detection system can obtain a unified and standardized input image, so that after inputting it into the target detection network model, the target detection result can be obtained quickly and accurately.

[0045] Since infrared images are obtained by measuring the heat radiated outward by an object, they have many advantages over visible light images, such as not being disturbed by environmental factors such as rain, snow, fog, and not being affected by lighting conditions. However, infrared images are also accompanied by problems such as poor resolution, low contrast, low signal-to-noise ratio, and missing texture features. Therefore, compared with the target detection method based on single infrared or visible light, the present application can comprehensively utilize the advantages of visible light and infrared images to more accurately extract features and enhance the detection capability of the target, especially in complex backgrounds, low contrast or occlusion. In addition, infrared and visible light dual-source data can provide a variety of feature expressions in target detection tasks. By fusing the features of the two, the target detection system can enhance performance and improve the generalization and robustness of the target detection network model.

[0046] 202. The shallow feature extraction module based on the target detection network model performs preliminary feature extraction on the visible light input image and the infrared input image to obtain a visible light shallow feature map and an infrared shallow feature map.

[0047] In the embodiment of the present application, the target detection system inputs the visible light input map and the infrared input map into the shallow feature extraction module of the target detection network model, performs preliminary feature extraction on the visible light input map and the infrared input map based on the shallow feature extraction module, and obtains the visible light shallow feature map and the infrared shallow feature map. The calculation formula is the following formula 1:

[0048] Formula 1:

[0049] Among them, H SF () represents the 3×3 convolutional layer calculation, x RGB represents the visible light input image, x IR represents the infrared input map, represents the shallow feature map of visible light, represents the infrared shallow feature map,

[0050] Subsequently, the visible light shallow feature map and the infrared shallow feature map are processed by the deep feature extraction module to obtain the visible light deep feature map and the infrared deep feature map. The calculation formula is as follows:

[0051] Formula 2:

[0052] Among them, H DF () represents a deep feature extraction module composed of residual Transformer layers. Each Transformer layer contains two attentions: spatial and channel. represents the shallow feature map of visible light, represents the infrared shallow feature map, represents the visible light deep feature map, Indicates infrared deep feature map like Figure 2B As shown in the figure, the residual network composed of the two-stream Transformer module first performs layer normalization on the shallow feature map, and then obtains the channel and spatial attention values ​​through channel and spatial attention calculation. The two attention values ​​are summed and layer normalized to obtain the feedforward neural network input feature map, which is then input into the two-stream feedforward neural network to obtain the deep feature map.

[0053] The deep feature extraction module proposed in this application includes a channel attention unit, a spatial attention unit, and a dual-stream feedforward neural network unit. Therefore, the specific process of inputting the visible light shallow feature map and the infrared shallow feature map into the deep feature extraction module to extract deep features is steps 203-207.

[0054] 203. Based on the deep feature extraction module, perform layer normalization operation on the visible light shallow feature map and the infrared shallow feature map to obtain a target feature map, obtain a preset number of divisions based on the deep feature extraction module, divide the target feature map according to the preset number of divisions, and obtain multiple sub-feature maps.

[0055] In the embodiment of the present application, the target detection system performs layer normalization operation on the visible light shallow feature map and the infrared shallow feature map based on the deep feature extraction module to obtain the target feature map x in ∈R 64×64×3 Next, the target detection system obtains the preset number of divisions based on the deep feature extraction module, divides the target feature map according to the preset number of divisions, and obtains multiple sub-feature maps x t ∈R 8×8×3 , where the preset number of divisions is 64. In this way, the target feature map after the layer normalization (LayerNormalization, LN) operation is divided into 64 sub-feature maps. Therefore, compared with the target detection method based on CNN, the present application can capture the global dependency of the input sequence in the process of processing long sequence input, so that the target detection system can better understand the context and related information of the entire image in processing the target detection task. At the same time, the self-attention mechanism of the target detection network model allows parallel computing, which can process input data more efficiently, thereby improving the training and detection speed of the model.

[0056] 204. Input the multiple sub-feature maps into the channel attention unit of the deep feature extraction module, and obtain the channel attention value corresponding to each sub-feature map output by the channel attention unit.

[0057] In the embodiment of the present application, the channel attention unit model is as follows Figure 2C As shown, the feature map is linearly mapped and reshaped into query value, index value, and content value through 1×1 point convolution and 3×3 depth convolution, where C×M 2 For 3×8 2 Then, the Softmax (regression) function is used to calculate the query value and the transposed index value to obtain the spectral attention map, and then the channel attention value is calculated using the content value and the spectral attention map.

[0058] Specifically, for each sub-feature map, the target detection system performs linear mapping on the sub-feature map based on the channel attention unit to obtain the target query value, target index value, and target content value corresponding to the sub-feature map. The calculation formula is as follows:

[0059] Formula 3:

[0060] in, represents the second 1×1 convolutional layer, represents the third 1×1 convolutional layer, represents the fourth 1×1 convolutional layer, represents the second 3×3 depth convolutional layer, represents the third 3×3 depth convolutional layer, represents the fourth 3×3 depth convolutional layer, x t represents the sub-feature map, Q channel represents the target query value, K channel Indicates the target index value, V channe l represents the target content value,

[0061] Next, the target detection system calculates the target query value and the target index value based on the channel attention unit to obtain the channel attention map corresponding to the sub-feature map. The calculation formula is as follows:

[0062] Formula 4:

[0063] Among them, A channel represents the channel attention map, Q channel represents the target query value, K channel Represents the target index value, T represents the transposition calculation, Softmax() represents the regression function, B represents the learnable relative position bias, and ε represents the learnable parameter.

[0064] Subsequently, the target detection system calculates the target content value and the channel attention map based on the channel attention unit to obtain the channel attention value corresponding to the sub-feature map. The calculation formula is as follows:

[0065] Formula 5: Attention(Q channel , K channel , V channel )=V channel ×A channel

[0066] Among them, Attention(Q channel , K channel , V channel ) represents the channel attention value, V channel Indicates the target content value, A channel Represents the channel attention map. Based on the above process, the target detection system can obtain the channel attention value of a sub-feature map.

[0067] Finally, the target detection system calculates each sub-feature map based on the channel attention unit to obtain the channel attention value corresponding to each sub-feature map. It should be noted that the self-attention mechanism of the target detection network model allows parallel calculation, so it can process input data more efficiently, thereby improving the detection speed of the model.

[0068] 205. Input the multiple sub-feature maps into the spatial attention unit of the deep feature extraction module, and obtain the spatial attention value corresponding to each sub-feature map output by the spatial attention unit.

[0069] In the embodiment of the present application, the spatial attention unit model is as follows: Figure 2D As shown in Figure 2, the feature map is linearly mapped to query value, index value, and content value through the projection matrix, where M 2 ×C is 8 2 ×3. Then, the Softmax (regression) function is used to calculate the query value and the transposed index value to obtain the spatial attention map, and then the spatial attention value is calculated using the content value and the spatial attention map.

[0070] Specifically, for each sub-feature map, the target detection system performs linear mapping on the sub-feature map based on the spatial attention unit to obtain the specified query value, specified index value, and specified content value corresponding to the sub-feature map. The calculation formula is as follows:

[0071] Formula 6

[0072] Among them, W Q represents the first learnable projection matrix, W K represents the second learnable projection matrix, W V represents the third learnable projection matrix, x t represents the sub-feature map, Q spa Indicates the specified query value. K spa Indicates the specified index value. V spa Indicates the specified content value.

[0073] Next, the target detection system calculates the specified query value and the specified index value based on the spatial attention unit to obtain the spatial attention map corresponding to the sub-feature map. The calculation formula is as follows:

[0074] Formula 7:

[0075] Among them, A spa represents the spatial attention map, Q spa Indicates the specified query value, K spaRepresents the specified index value, Softmax() represents the regression function, B represents the learnable relative position bias, and C represents the number of channels, which is 3.

[0076] Subsequently, the target detection system calculates the spatial attention map and the specified content value based on the spatial attention unit to obtain the spatial attention value corresponding to the sub-feature map. The calculation formula is as follows 8:

[0077] Formula 8: Attention(Q spa , K spa , V spa )=V spa ×A spa

[0078] Among them, Attention(Q spa , K spa , V spa ) represents the spatial attention value, V spa Indicates the specified content value, A spa Represents the spatial attention map. Based on the above process, the target detection system can obtain the spatial attention value of a sub-feature map.

[0079] Based on the spatial attention unit, each sub-feature map is calculated to obtain the spatial attention value corresponding to each sub-feature map. It should be noted that the self-attention mechanism of the target detection network model allows parallel calculation, so the input data can be processed more efficiently, thereby improving the detection speed of the model.

[0080] 206. Based on the deep feature extraction module, the channel attention value and the spatial attention value corresponding to each sub-feature map are summed up to obtain the target attention value corresponding to each sub-feature map. Based on the deep feature extraction module, layer normalization operations are performed on the target attention values ​​corresponding to each sub-feature map to obtain multiple feedforward neural network input feature maps.

[0081] In the embodiment of the present application, the target detection system calculates the sum of the channel attention value and the spatial attention value corresponding to each sub-feature map based on the deep feature extraction module to obtain the target attention value corresponding to each sub-feature map. Then, the target detection system performs layer normalization operation on the target attention value corresponding to each sub-feature map based on the deep feature extraction module to obtain multiple feedforward neural network input feature maps.

[0082] Compared with the original Transformer-based object detection method, the deep feature extraction module considers both channel and spatial attention, so that the object detection network model can simultaneously enhance the representation ability of target features in the channel dimension and the spatial dimension, thereby improving the accuracy and robustness of object detection. At the same time, the introduction of the channel attention mechanism can adaptively adjust the weights of feature channels, making the model more sensitive to objects of different sizes. Therefore, by combining channel attention and spatial attention, the object detection network model can better adapt to objects of different sizes and improve the generalization ability of object detection.

[0083] 207. Input the multiple feedforward neural network input feature maps into the dual-stream feedforward neural network unit of the deep feature extraction module to obtain a visible light deep feature map and an infrared deep feature map.

[0084] In the embodiment of the present application, the target detection system inputs multiple feedforward neural network input feature maps into the dual-stream feedforward neural network unit of the deep feature extraction module to obtain a visible light deep feature map and an infrared deep feature map, and the calculation formula is the following formula 9:

[0085]

[0086] Formula 9:

[0087] Among them, x″ represents the deep feature map, which includes the visible light deep feature map and infrared deep feature map represents the input feature map of the feedforward neural network, LN() represents the layer normalization calculation, W 1 represents the first fully connected layer, W 2 represents the second fully connected layer, W P represents the first 1×1 convolutional layer, W D represents the first 3×3 depth convolutional layer, H Gelu () represents the activation function. It should be noted that the two-stream feedforward neural network unit model is as follows Figure 2E As shown, it is equivalent to the calculation process of formula 9. This application can solve the limitations of traditional methods and CNN-based methods in the field of target detection. This application uses Transformer to expand the perception domain of the network, so that the target detection system can parse global information and effectively enhance the representation ability of the network. In addition, the introduction of dual-stream attention can not only explore the local and non-local similarities of space, but also take into account the connection between space and channels, thereby achieving synergistic enhancement of features.

[0088] The feature fusion module based on the target detection network model aggregates the visible light input map, the infrared input map, the visible light shallow feature map, the infrared shallow feature map, the visible light deep feature map, and the infrared deep feature map to obtain a visible light feature map and an infrared feature map.

[0089] In an embodiment of the present application, the target detection system inputs the visible light input map, the infrared input map, the visible light shallow feature map, the infrared shallow feature map, the visible light deep feature map, and the infrared deep feature map into a feature fusion module of the target detection network model, and aggregates the visible light input map, the infrared input map, the visible light shallow feature map, the infrared shallow feature map, the visible light deep feature map, and the infrared deep feature map based on the feature fusion module to obtain a visible light feature map and an infrared feature map, and the calculation formula is the following formula 10:

[0090] Formula 10:

[0091] in, represents the visible light characteristic map, represents the infrared characteristic image, x RGB represents the visible light input image, x IR represents the infrared input map, represents the shallow feature map of visible light, represents the infrared shallow feature map, represents the visible light deep feature map, represents the infrared deep feature map, H EN () represents the feature enhancement algorithm. The target detection system achieves high-quality information fusion by aggregating deep feature maps, shallow feature maps and original images, and can retain more details, thereby improving the accuracy of target detection.

[0092] 209. The detection head based on the target detection network model predicts the visible light feature map and the infrared feature map to obtain the target detection result, and pushes the target detection result to the user who initiated the target detection request.

[0093] In the embodiment of the present application, the target detection system inputs the visible light feature map and the infrared feature map into the detection head of the target detection network model, and performs target prediction on the visible light feature map and the infrared feature map based on the detection head to obtain the position information and category information of the target object. Then, the target detection system generates the target detection result using the position information and category information of the target object, and the calculation formula is the following formula 11:

[0094] Formula 11:

[0095] Where I represents the target image feature map, which includes the visible light feature map and infrared signature Boundingbox represents the location information of the target object, class represents the category information of the target object, and MLP() represents a multi-layer perceptron, which acts as the detection head of the target detection network model. It makes two predictions for each query object, predicting the location information and category information of the target, and obtains the predicted box information, which is used as the target detection result.

[0096] In an optional implementation, the target detection system reads the predicted box position corresponding to the target object in the target detection result, and obtains the real box position corresponding to the target object in the database. Then, the target detection system obtains the L1 smooth loss function, and uses the L1 smooth loss function to calculate the predicted box position and the real box position to obtain the loss function. Then, the target detection system uses the loss function to calculate the target network parameters, and uses the target network parameters to adjust the target detection network model, thereby improving the accuracy of the target detection network model.

[0097] In summary, this application proposes a network flow diagram of visible light and infrared image target detection based on dual-stream attention as follows:

[0098] like Figure 2F , 2G As shown in the figure, the input visible light and infrared images are first aligned, and the shallow feature map is preliminarily extracted using the 3×3 convolutional layer of the shallow feature extraction module. Then, the obtained shallow feature map is input into the deep feature extraction module composed of the two-stream attention Transformer layer to extract deep features, where the deep feature extraction module includes channel attention, spatial attention, and a two-stream feedforward neural network. Subsequently, the feature fusion module is used to aggregate deep features, shallow features, and original images to achieve high-quality information fusion. Finally, the fused feature map is input into the multi-layer perceptron as the detection head to solve the problem of image detail information loss and edge extraction fuzziness, thereby achieving accurate and robust target detection effects for dual tasks.

[0099] The method provided in the embodiment of the present application, in response to a target detection request, obtains a visible light image and an infrared image carried in the target detection request, performs data alignment operations on the visible light image and the infrared image to obtain a visible light input map and an infrared input map, performs preliminary feature extraction on the visible light input map and the infrared input map based on a shallow feature extraction module of a target detection network model to obtain a visible light shallow feature map and an infrared shallow feature map, performs deep feature extraction on the visible light shallow feature map and the infrared shallow feature map based on a deep feature extraction module of a target detection network model to obtain a visible light deep feature map and an infrared deep feature map, a feature fusion module based on the target detection network model aggregates the visible light input map, the infrared input map, the visible light shallow feature map, the infrared shallow feature map, the visible light deep feature map, and the infrared deep feature map to obtain a visible light feature map and an infrared feature map, a detection head based on the target detection network model predicts the visible light feature map and the infrared feature map to obtain a target detection result, and pushes the target detection result to a user who initiates the target detection request, wherein the deep feature extraction module includes a channel attention unit, a spatial attention unit, and a dual-stream feedforward neural network unit. This application performs data alignment of the input visible light and infrared images, and then uses a 3×3 convolutional layer to preliminarily extract shallow feature maps from the processed data, and inputs the obtained shallow feature maps into a residual network composed of a two-stream Transformer module to extract deep feature maps, and then aggregates the deep feature maps, shallow feature maps, and original images, and inputs the aggregated feature maps into the detection head to achieve target detection. This application considers both channel attention and spatial attention, so that the target detection network model can not only enhance the representation capabilities of target features in the channel dimension and spatial dimension at the same time, thereby improving the accuracy and robustness of target detection, but also better adapt to targets of different sizes and improve the generalization ability of target detection. Moreover, this application comprehensively utilizes the advantages of visible light images and infrared images, can accurately extract features, improve the generalization ability and robustness of the target detection network model, and thus achieve accurate target detection in complex backgrounds, low contrast or occlusion conditions.

[0100] Further, as Figure 1 The specific implementation of the method, the embodiment of the present application provides a visible light and infrared image target detection device based on dual-stream attention, such as Figure 3A As shown, the device includes: a preprocessing module 301, a feature extraction module 302, a feature aggregation module 303 and a feature detection module 304.

[0101] The preprocessing module 301 is used to respond to a target detection request, obtain a visible light image and an infrared image carried in the target detection request, perform a data alignment operation on the visible light image and the infrared image, and obtain a visible light input image and an infrared input image;

[0102] A feature extraction module 302 is used to perform preliminary feature extraction on the visible light input map and the infrared input map based on the shallow feature extraction module of the target detection network model to obtain a visible light shallow feature map and an infrared shallow feature map, and to perform deep feature extraction on the visible light shallow feature map and the infrared shallow feature map based on the deep feature extraction module of the target detection network model to obtain a visible light deep feature map and an infrared deep feature map, wherein the deep feature extraction module includes a channel attention unit, a spatial attention unit, and a dual-stream feedforward neural network unit;

[0103] A feature aggregation module 303 is used to aggregate the visible light input map, the infrared input map, the visible light shallow feature map, the infrared shallow feature map, the visible light deep feature map, and the infrared deep feature map based on the feature fusion module of the target detection network model to obtain a visible light feature map and an infrared feature map;

[0104] The feature detection module 304 is used to predict the visible light feature map and the infrared feature map based on the detection head of the target detection network model to obtain a target detection result, and push the target detection result to the user who initiated the target detection request.

[0105] In a specific application scenario, the preprocessing module 301 is used to obtain a preset spatial resolution, segment the visible light image according to the preset spatial resolution, and obtain the visible light input image; segment the infrared image according to the preset spatial resolution to obtain the segmented infrared image, obtain a channel interpolation algorithm and a preset number of channels, and use the channel interpolation algorithm to expand the segmented infrared image according to the preset number of channels to obtain the infrared input image.

[0106] In a specific application scenario, the feature extraction module 302 is used to input the visible light input map and the infrared input map into the shallow feature extraction module of the target detection network model, and perform preliminary feature extraction on the visible light input map and the infrared input map based on the shallow feature extraction module to obtain the visible light shallow feature map and the infrared shallow feature map.

[0107]

[0108] Among them, H SF O represents the 3×3 convolution layer calculation, x RGB represents the visible light input map, X IR represents the infrared input map, represents the visible light shallow feature map, Represents the infrared shallow layer feature map.

[0109] In a specific application scenario, the feature extraction module 302 is used to perform layer normalization operations on the visible light shallow feature map and the infrared shallow feature map based on the deep feature extraction module to obtain a target feature map; obtain a preset number of divisions based on the deep feature extraction module, divide the target feature map according to the preset number of divisions to obtain a plurality of sub-feature maps; input the plurality of sub-feature maps into the channel attention unit of the deep feature extraction module, obtain the channel attention value corresponding to each of the sub-feature maps output by the channel attention unit, and input the plurality of sub-feature maps into the spatial attention unit of the deep feature extraction module. element, obtaining the spatial attention value corresponding to each of the sub-feature maps output by the spatial attention unit; based on the deep feature extraction module, summing up the channel attention value and the spatial attention value corresponding to each of the sub-feature maps to obtain the target attention value corresponding to each of the sub-feature maps; based on the deep feature extraction module, performing layer normalization operation on the target attention value corresponding to each of the sub-feature maps to obtain a plurality of feedforward neural network input feature maps, inputting the plurality of feedforward neural network input feature maps into the dual-stream feedforward neural network unit of the deep feature extraction module to obtain the visible light deep feature map and the infrared deep feature map,

[0110]

[0111] Wherein, X″ represents a deep feature map, and the deep feature map includes the visible light deep feature map And the infrared deep feature map represents the feedforward neural network input feature map, LN() represents the layer normalization calculation, W 1 represents the first fully connected layer, W 2 represents the second fully connected layer, W P represents the first 1×1 convolutional layer, W D represents the first 3×3 depth convolutional layer, H Gelu () represents the activation function.

[0112] In a specific application scenario, the feature extraction module 302 is used to perform linear mapping on each of the sub-feature graphs based on the channel attention unit to obtain a target query value, a target index value, and a target content value corresponding to the sub-feature graph.

[0113]

[0114] in, represents the second 1×1 convolutional layer, represents the third 1×1 convolutional layer, represents the fourth 1×1 convolutional layer, represents the second 3×3 depth convolutional layer, represents the third 3×3 depth convolutional layer, represents the fourth 3×3 depth convolutional layer, x t represents the sub-feature map, Q channel represents the target query value, K channel represents the target index value, V channel represents the target content value; based on the channel attention unit, the target query value and the target index value are calculated to obtain a channel attention map corresponding to the sub-feature map,

[0115]

[0116] Among them, A channel represents the channel attention map, Q channel represents the target query value, K channel represents the target index value, T represents transposition calculation, Softmax() represents regression function, B represents learnable relative position bias, and ε represents learnable parameter; based on the channel attention unit, the target content value and the channel attention map are calculated to obtain the channel attention value corresponding to the sub-feature map,

[0117] Attention(Q channel , K channel , V channel )=V channel ×A channel

[0118] Among them, Attention(Q channel , K channel , V channel ) represents the channel attention value, V channel Represents the target content value, A channel Representing the channel attention map; calculating each of the sub-feature maps based on the channel attention unit to obtain the channel attention value corresponding to each of the sub-feature maps.

[0119] In a specific application scenario, the feature extraction module 302 is used to perform linear mapping on each of the sub-feature graphs based on the spatial attention unit to obtain a specified query value, a specified index value, and a specified content value corresponding to the sub-feature graph.

[0120]

[0121] Among them, W Q represents the first learnable projection matrix, W K represents the second learnable projection matrix, W Vrepresents the third learnable projection matrix, x t represents the sub-feature map, Q spa represents the specified query value, K spa Indicates the specified index value, V spa represents the specified content value; based on the spatial attention unit, the specified query value and the specified index value are calculated to obtain the spatial attention map corresponding to the sub-feature map,

[0122]

[0123] Among them, A spa represents the spatial attention map, Q spa represents the specified query value, K spa represents the specified index value, Softmax() represents the regression function, B represents the learnable relative position bias, C represents the number of channels, and the value is 3; based on the spatial attention unit, the spatial attention map and the specified content value are calculated to obtain the spatial attention value corresponding to the sub-feature map,

[0124] Attention(Q spa , K spa , V spa )=V spa ×A spa

[0125] Among them, Attention(Q spa , K spa , V spa ) represents the spatial attention value, V spa Indicates the specified content value, A spa Representing the spatial attention map; calculating each of the sub-feature maps based on the spatial attention unit to obtain a spatial attention value corresponding to each of the sub-feature maps.

[0126] In a specific application scenario, the feature aggregation module 303 is used to input the visible light input map, the infrared input map, the visible light shallow feature map, the infrared shallow feature map, the visible light deep feature map, and the infrared deep feature map into the feature fusion module of the target detection network model, and aggregate the visible light input map, the infrared input map, the visible light shallow feature map, the infrared shallow feature map, the visible light deep feature map, and the infrared deep feature map based on the feature fusion module to obtain the visible light feature map and the infrared feature map.

[0127]

[0128] in, represents the visible light characteristic graph, represents the infrared characteristic image, x RGB represents the visible light input map, X IR represents the infrared input map, represents the visible light shallow feature map, represents the infrared shallow layer feature map, represents the visible light deep feature map, represents the infrared deep feature map, H EN () represents the feature enhancement algorithm.

[0129] In a specific application scenario, the feature detection module 304 is used to input the visible light feature map and the infrared feature map into the detection head of the target detection network model, perform target prediction on the visible light feature map and the infrared feature map based on the detection head, obtain the position information of the target object and the category information of the target object, and generate the target detection result using the position information of the target object and the category information of the target object.

[0130] class = MLP(I)

[0131] boundingbox=MLP(I)

[0132] Wherein, I represents the target image feature map, and the target image feature map includes the visible light feature map And the infrared characteristic map boundingbox represents the location information of the target object, class represents the category information of the target object, and MLP() represents a multi-layer perceptron.

[0133] In specific application scenarios, such as Figure 3B As shown, the device also includes: a model updating module 305.

[0134] The model updating module 305 is used to read the predicted box position corresponding to the target object in the target detection result, and obtain the real box position corresponding to the target object in the database; obtain an L1 smooth loss function, and use the L1 smooth loss function to calculate the predicted box position and the real box position to obtain a loss function; use the loss function to calculate the target network parameters, and use the target network parameters to adjust the target detection network model.

[0135] The device provided in the embodiment of the present application, in response to a target detection request, obtains a visible light image and an infrared image carried in the target detection request, performs data alignment operations on the visible light image and the infrared image to obtain a visible light input map and an infrared input map, performs preliminary feature extraction on the visible light input map and the infrared input map based on a shallow feature extraction module of a target detection network model to obtain a visible light shallow feature map and an infrared shallow feature map, performs deep feature extraction on the visible light shallow feature map and the infrared shallow feature map based on a deep feature extraction module of a target detection network model to obtain a visible light deep feature map and an infrared deep feature map, a feature fusion module based on the target detection network model aggregates the visible light input map, the infrared input map, the visible light shallow feature map, the infrared shallow feature map, the visible light deep feature map, and the infrared deep feature map to obtain a visible light feature map and an infrared feature map, a detection head based on the target detection network model predicts the visible light feature map and the infrared feature map to obtain a target detection result, and pushes the target detection result to a user who initiates the target detection request, wherein the deep feature extraction module includes a channel attention unit, a spatial attention unit, and a dual-stream feedforward neural network unit. This application performs data alignment of the input visible light and infrared images, and then uses a 3×3 convolutional layer to preliminarily extract shallow feature maps from the processed data, and inputs the obtained shallow feature maps into a residual network composed of a two-stream Transformer module to extract deep feature maps, and then aggregates the deep feature maps, shallow feature maps, and original images, and inputs the aggregated feature maps into the detection head to achieve target detection. This application considers both channel attention and spatial attention, so that the target detection network model can not only enhance the representation capabilities of target features in the channel dimension and spatial dimension at the same time, thereby improving the accuracy and robustness of target detection, but also better adapt to targets of different sizes and improve the generalization ability of target detection. Moreover, this application comprehensively utilizes the advantages of visible light images and infrared images, can accurately extract features, improve the generalization ability and robustness of the target detection network model, and thus achieve accurate target detection in complex backgrounds, low contrast or occlusion conditions.

[0136] It should be noted that for other corresponding descriptions of the functional units involved in the visible light and infrared image target detection device based on dual-stream attention provided in the embodiment of the present application, please refer to Figure 1 and Figures 2A to 2G The corresponding description in will not be repeated here.

[0137] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0138] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

[0140] In an exemplary embodiment, see Figure 4 , and also provides a computer device, which includes a bus, a processor, a memory and a communication interface, and may also include an input and output interface and a display device, wherein each functional unit can communicate with each other through the bus. The memory stores a computer program, and the processor is used to execute the program stored in the memory to execute the visible light and infrared image target detection method based on dual-stream attention in the above embodiment.

[0141] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for detecting targets in visible light and infrared images based on dual-stream attention.

[0142] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by hardware, or by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0143] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred implementation scenario, and the modules or processes in the accompanying drawing are not necessarily required for implementing the present application.

[0144] Those skilled in the art will appreciate that the modules in the device in the implementation scenario can be distributed in the device in the implementation scenario according to the implementation scenario description, or can be changed accordingly and located in one or more devices different from the implementation scenario. The modules in the above implementation scenario can be combined into one module, or can be further split into multiple sub-modules.

[0145] The above application serial numbers are for description only and do not represent the advantages or disadvantages of the implementation scenarios.

[0146] The above disclosure only discloses several specific implementation scenarios of the present application. However, the present application is not limited thereto, and any changes that can be conceived by technicians in this field should fall within the scope of protection of the present application.

Claims

1. A method for target detection in visible light and infrared images based on dual-stream attention, characterized in that: include: In response to a target detection request, obtaining a visible light image and an infrared image carried in the target detection request, performing a data alignment operation on the visible light image and the infrared image to obtain a visible light input image and an infrared input image; A shallow feature extraction module based on the target detection network model performs preliminary feature extraction on the visible light input map and the infrared input map to obtain a shallow visible light feature map and an infrared shallow feature map; a deep feature extraction module based on the target detection network model performs deep feature extraction on the shallow visible light feature map and the infrared shallow feature map to obtain a deep visible light feature map and an infrared deep feature map, wherein the deep feature extraction module includes a channel attention unit, a spatial attention unit, and a dual-stream feedforward neural network unit; A feature fusion module based on the target detection network model aggregates the visible light input map, the infrared input map, the visible light shallow feature map, the infrared shallow feature map, the visible light deep feature map, and the infrared deep feature map to obtain a visible light feature map and an infrared feature map; The detection head based on the target detection network model predicts the visible light feature map and the infrared feature map to obtain a target detection result, and pushes the target detection result to the user who initiated the target detection request; The deep feature extraction module based on the target detection network model performs deep feature extraction on the visible light shallow feature map and the infrared shallow feature map to obtain a visible light deep feature map and an infrared deep feature map, including: Based on the deep feature extraction module, a layer normalization operation is performed on the visible light shallow feature map and the infrared shallow feature map to obtain a target feature map; Acquire a preset number of divisions based on the deep feature extraction module, and divide the target feature map according to the preset number of divisions to obtain a plurality of sub-feature maps; For each of the sub-feature graphs, linear mapping is performed on the sub-feature graph based on the channel attention unit to obtain a target query value, a target index value, and a target content value corresponding to the sub-feature graph. in, represents the second 1×1 convolutional layer, represents the third 1×1 convolutional layer, represents the fourth 1×1 convolutional layer, represents the second 3×3 depth convolutional layer, represents the third 3×3 depth convolutional layer, represents the fourth 3×3 depth convolutional layer, x t represents the sub-feature map, Q channel represents the target query value, K channel represents the target index value, V channel represents the target content value; The target query value and the target index value are calculated based on the channel attention unit to obtain a channel attention map corresponding to the sub-feature map, Among them, A channel represents the channel attention map, Q channel represents the target query value, K channel represents the target index value, T represents the transposition calculation, Softmax() represents the regression function, B represents the learnable relative position bias, and ε represents the learnable parameter; The target content value and the channel attention map are calculated based on the channel attention unit to obtain a channel attention value corresponding to the sub-feature map. Attention(Q channel ,K channel ,V channel )=V channel ×A channel Among them, Attention(Q channel , K channel , V channel ) represents the channel attention value, V channel Represents the target content value, A channel represents the channel attention map; Calculating each of the sub-feature maps based on the channel attention unit to obtain a channel attention value corresponding to each of the sub-feature maps; Input the multiple sub-feature maps into the spatial attention unit of the deep feature extraction module, and obtain the spatial attention value corresponding to each of the sub-feature maps output by the spatial attention unit; Based on the deep feature extraction module, the channel attention value and the spatial attention value corresponding to each of the sub-feature maps are summed up to obtain a target attention value corresponding to each of the sub-feature maps; Based on the deep feature extraction module, a layer normalization operation is performed on the target attention value corresponding to each of the sub-feature maps to obtain a plurality of feedforward neural network input feature maps, and the plurality of feedforward neural network input feature maps are input into the dual-stream feedforward neural network unit of the deep feature extraction module to obtain the visible light deep feature map and the infrared deep feature map. Wherein, x″ represents a deep feature map, and the deep feature map includes the visible light deep feature map And the infrared deep feature map represents the feedforward neural network input feature map, LN() represents the layer normalization calculation, W 1 represents the first fully connected layer, W 2 represents the second fully connected layer, W P represents the first 1×1 convolutional layer, W D represents the first 3×3 depth convolutional layer, H Gelu () represents the activation function.

2. The method according to claim 1, characterized in that The performing of data alignment operation on the visible light image and the infrared image to obtain a visible light input image and an infrared input image includes: Acquire a preset spatial resolution, and segment the visible light image according to the preset spatial resolution to obtain the visible light input image; The infrared image is segmented according to the preset spatial resolution to obtain the segmented infrared image, a channel interpolation algorithm and a preset number of channels are acquired, and the segmented infrared image is expanded according to the preset number of channels using the channel interpolation algorithm to obtain the infrared input image.

3. The method according to claim 1, characterized in that: The shallow feature extraction module based on the target detection network model performs preliminary feature extraction on the visible light input map and the infrared input map to obtain a visible light shallow feature map and an infrared shallow feature map, including: Inputting the visible light input map and the infrared input map into the shallow feature extraction module of the target detection network model, performing preliminary feature extraction on the visible light input map and the infrared input map based on the shallow feature extraction module to obtain the visible light shallow feature map and the infrared shallow feature map, Among them, H SF () represents the 3×3 convolutional layer calculation, x RGB represents the visible light input image, x IR represents the infrared input map, represents the visible light shallow feature map, Represents the infrared shallow layer feature map.

4. The method according to claim 1, characterized in that: The step of inputting the plurality of sub-feature maps into the spatial attention unit of the deep feature extraction module, and obtaining a spatial attention value corresponding to each of the sub-feature maps output by the spatial attention unit, comprises: For each of the sub-feature graphs, linear mapping is performed on the sub-feature graph based on the spatial attention unit to obtain a specified query value, a specified index value, and a specified content value corresponding to the sub-feature graph, Q spa =W Q x t ,K apa =W K x t ,V spa =W V x t Among them, W Q represents the first learnable projection matrix, W K represents the second learnable projection matrix, W V represents the third learnable projection matrix, x t represents the sub-feature map, Q spa represents the specified query value, K spa Indicates the specified index value, V spa Represents the specified content value; The specified query value and the specified index value are calculated based on the spatial attention unit to obtain a spatial attention map corresponding to the sub-feature map, Among them, A spa represents the spatial attention map, Q spa represents the specified query value, K spa represents the specified index value, Softmax() represents the regression function, B represents the learnable relative position bias, and C represents the number of channels, which is 3; The spatial attention map and the specified content value are calculated based on the spatial attention unit to obtain a spatial attention value corresponding to the sub-feature map. Attention(Q spa ,K spa ,V spa )=V spa ×A spa Among them, Attention(Q spa , K spa , V spa ) represents the spatial attention value, V spa Indicates the specified content value, A spa represents the spatial attention map; Based on the spatial attention unit, each of the sub-feature maps is calculated to obtain a spatial attention value corresponding to each of the sub-feature maps.

5. The method according to claim 1, characterized in that The feature fusion module based on the target detection network model aggregates the visible light input map, the infrared input map, the visible light shallow feature map, the infrared shallow feature map, the visible light deep feature map, and the infrared deep feature map to obtain a visible light feature map and an infrared feature map, including: The visible light input map, the infrared input map, the visible light shallow feature map, the infrared shallow feature map, the visible light deep feature map, and the infrared deep feature map are input into a feature fusion module of the target detection network model; based on the feature fusion module, the visible light input map, the infrared input map, the visible light shallow feature map, the infrared shallow feature map, the visible light deep feature map, and the infrared deep feature map are aggregated to obtain the visible light feature map and the infrared feature map; in, represents the visible light characteristic graph, represents the infrared characteristic image, x RGB represents the visible light input image, x IR represents the infrared input map, represents the visible light shallow feature map, represents the infrared shallow layer feature map, represents the visible light deep feature map, represents the infrared deep feature map, H EN () represents the feature enhancement algorithm.

6. The method according to claim 1, characterized in that The detection head based on the target detection network model predicts the visible light feature map and the infrared feature map to obtain a target detection result, including: The visible light feature map and the infrared feature map are input into the detection head of the target detection network model, and the target is predicted based on the detection head to obtain the position information and the category information of the target object, and the target detection result is generated by using the position information and the category information of the target object. class = MLP(I) boundingbox=MLP(I) Wherein, I represents the target image feature map, and the target image feature map includes the visible light feature map And the infrared characteristic map boundingbox represents the location information of the target object, class represents the category information of the target object, and MLP() represents a multi-layer perceptron.

7. The method according to claim 1, characterized in that The method further comprises: Reading the predicted frame position corresponding to the target object in the target detection result, and obtaining the real frame position corresponding to the target object in the database; Obtain an L1 smooth loss function, and use the L1 smooth loss function to calculate the predicted box position and the real box position to obtain a loss function; The target network parameters are calculated using the loss function, and the target network parameters are used to adjust the target detection network model.

8. A visible light and infrared image target detection device based on dual-stream attention, characterized in that: include: a preprocessing module, configured to, in response to a target detection request, obtain a visible light image and an infrared image carried in the target detection request, perform a data alignment operation on the visible light image and the infrared image, and obtain a visible light input image and an infrared input image; A feature extraction module, which is used to perform preliminary feature extraction on the visible light input map and the infrared input map based on a shallow feature extraction module of a target detection network model to obtain a visible light shallow feature map and an infrared shallow feature map, and to perform deep feature extraction on the visible light shallow feature map and the infrared shallow feature map based on a deep feature extraction module of the target detection network model to obtain a visible light deep feature map and an infrared deep feature map, wherein the deep feature extraction module includes a channel attention unit, a spatial attention unit, and a dual-stream feedforward neural network unit; A feature aggregation module, configured to aggregate the visible light input map, the infrared input map, the visible light shallow feature map, the infrared shallow feature map, the visible light deep feature map, and the infrared deep feature map based on the feature fusion module of the target detection network model to obtain a visible light feature map and an infrared feature map; A feature detection module, used to predict the visible light feature map and the infrared feature map based on the detection head of the target detection network model, obtain a target detection result, and push the target detection result to the user who initiated the target detection request; The feature extraction module is further used to perform layer normalization operation on the visible light shallow feature map and the infrared shallow feature map based on the deep feature extraction module to obtain a target feature map; obtain a preset number of divisions based on the deep feature extraction module, divide the target feature map according to the preset number of divisions to obtain multiple sub-feature maps; for each of the sub-feature maps, linearly map the sub-feature map based on the channel attention unit to obtain a target query value, a target index value, and a target content value corresponding to the sub-feature map. in, represents the second 1×1 convolutional layer, represents the third 1×1 convolutional layer, represents the fourth 1×1 convolutional layer, represents the second 3×3 depth convolutional layer, represents the third 3×3 depth convolutional layer, represents the fourth 3×3 depth convolutional layer, x t represents the sub-feature map, Q channel represents the target query value, K channel represents the target index value, V channel represents the target content value; based on the channel attention unit, the target query value and the target index value are calculated to obtain a channel attention map corresponding to the sub-feature map, Among them, A channel represents the channel attention map, Q channel represents the target query value, K channel represents the target index value, T represents transposition calculation, Softmax() represents regression function, B represents learnable relative position bias, and ε represents learnable parameter; based on the channel attention unit, the target content value and the channel attention map are calculated to obtain the channel attention value corresponding to the sub-feature map, Attention(Q channel ,L channel ,V channel )=V ch annel×A channel Among them, Attention(Q channel , K channel , V channel ) represents the channel attention value, V channel Represents the target content value, A channel Representing the channel attention map; calculating each of the sub-feature maps based on the channel attention unit to obtain the channel attention value corresponding to each of the sub-feature maps; inputting the multiple sub-feature maps into the spatial attention unit of the deep feature extraction module to obtain the spatial attention value corresponding to each of the sub-feature maps output by the spatial attention unit; summing the channel attention value and the spatial attention value corresponding to each of the sub-feature maps based on the deep feature extraction module to obtain the target attention value corresponding to each of the sub-feature maps; performing layer normalization operation on the target attention value corresponding to each of the sub-feature maps based on the deep feature extraction module to obtain multiple feedforward neural network input feature maps, inputting the multiple feedforward neural network input feature maps into the dual-stream feedforward neural network unit of the deep feature extraction module to obtain the visible light deep feature map and the infrared deep feature map, Wherein, x″ represents a deep feature map, and the deep feature map includes the visible light deep feature map And the infrared deep feature map represents the feedforward neural network input feature map, LN() represents the layer normalization calculation, W 1 represents the first fully connected layer, W 2 represents the second fully connected layer, W P represents the first 1×1 convolutional layer, W D represents the first 3×3 depth convolutional layer, H Gelu () represents the activation function.

Images