Bimodal object detection model and method based on residual shrinkage attention network

Abstract

The purpose of the present invention is to provide a dual-modal target detection model and method based on a residual shrinkage attention network. The designed model includes: a dual-stream feature extraction network, a dual-modal feature fusion module and a multi-scale path aggregation module. Among them, the dual-stream feature extraction module performs multi-scale feature extraction on the input infrared and visible light images; the dual-modal feature fusion module, based on the residual shrinkage attention network, performs adaptive soft thresholding denoising and fusion on the dual-modal features to obtain more significant fusion features of the target features; the multi-scale path aggregation module obtains multi-scale features that aggregate position information and semantic information by cascading four fusion features of different scales and combining them with the position attention module, and sends them to the three-layer detection module to obtain the detection results. Through the method of the present invention, by utilizing the features of infrared and visible light images, more significant fusion features of target information can be obtained in low-light scenes at night, thereby improving the target detection effect.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and image processing, and in particular to a dual-modal target detection model and method based on a residual shrinkage attention network. Background Art

[0002] Object detection is a fundamental task in computer vision, playing a crucial role in video surveillance, pedestrian recognition, and intelligent driving. Traditional object detection methods typically involve sliding windows of varying sizes across an image to obtain candidate regions. These regions are then classified and judged by a classifier in traditional machine learning based on their visual features. However, the region selection strategy based on sliding windows is manually designed, highly complex, and difficult to adapt to diverse scene targets. With the development of deep learning, convolutional neural networks (CNNs) perform translation calculations on images by initializing convolution kernels to obtain corresponding feature maps. The convolution kernels are continuously adjusted during backpropagation during training to better extract feature maps. This makes the CNN framework highly adaptable to diverse scene targets, and as a result, it has quickly become the mainstream approach for object detection algorithms.

[0003] Currently, commonly used target detection algorithms primarily use single-modal images as training data, but using only single-modal images yields poor detection results when faced with complex scenes. Images acquired by visible light cameras typically have high spatial resolution and rich texture details, consistent with human visual perception. However, visible light images are easily affected by changes in illumination, and target features are easily lost in low-light environments at night. Infrared cameras, which reflect the thermal radiation information of objects, can resist interference caused by changes in illumination, but they typically have lower resolution and poorer texture details. Therefore, for targets in complex scenes such as low illumination, using only single-modal images of infrared or visible light is ineffective and difficult to cope with changes in complex scenes. Therefore, some scholars have focused on detection methods that combine dual-modal images, attempting to exploit information between infrared and visible light modalities through different fusion strategies to improve the performance of target detection algorithms in complex environments.

[0004] The patent document with application number (CN202110878261.3) proposes to pass visible light and infrared images into parallel pre-trained feature extraction networks respectively to extract feature maps of three different scales; use conditional generative adversarial networks to fuse multimodal and multi-scale feature maps, so that more target feature information is injected into the feature map of the original visible light modality; cascade the multi-scale and multi-modal fusion feature maps in sequence and pass them to each YOLO layer to complete the target detection task, thereby improving the detection effect of the target detection model in night environments.

[0005] The patent document with application number (CN202110475048.8) proposes to build a lightweight illumination perception network to realize the calculation of visible light modal image perception coefficients, and perform multimodal fusion of infrared modal and visible light modal data under the YOLOv5 architecture. This method achieves good detection results in complex environment data sets such as multimodal night scenes.

[0006] Currently, due to the inherent limitations of infrared and visible light imaging, images captured in complex, low-light scenes at night often suffer from significant background noise and unclear target features. Directly fusing these images introduces unnecessary redundant information, making it difficult to generate effective feature maps. Consequently, this approach may be incapable of detecting objects in complex, low-light scenes. Furthermore, current dual-modal object detection methods fail to fully exploit the multi-scale nature of features, limiting the network's ability to detect smaller objects. Therefore, addressing these challenges is crucial. Summary of the Invention

[0007] In response to the problems and defects of the existing technology, the present invention proposes a model and method for dual-modal target detection based on a residual shrinkage attention network to improve detection performance in low-light environments at night. The present invention proposes a model for dual-modal target detection, which uses a residual shrinkage attention network to focus on target features under complex background interference, remove irrelevant noise and background redundancy, and efficiently fuse modal features of different scales; it uses multi-scale path aggregation to add an additional layer of shallow features, combined with a position attention module, to fully utilize shallow position information and deep semantic information, to overcome the problems of existing models that are difficult to obtain obvious target features when the source image quality is poor and detection failure in different scenes, and to improve the algorithm's target detection effect in complex scenes with low light at night.

[0008] The designed model comprises a dual-stream feature extraction network, a dual-modal feature fusion module, and a multi-scale path aggregation module. The dual-stream feature extraction module extracts multi-scale features from the input infrared and visible light images. The dual-modal feature fusion module, based on a residual shrinkage attention network, adaptively soft-thresholds the dual-modal features to denoise and fuse them, generating multi-scale fused features. The multi-scale path aggregation module cascades fused features from four different scales to generate multi-scale features that aggregate positional and semantic information. These features are then fed into a three-layer detection module to obtain detection results. The dual-modal feature fusion module uses adaptive residual attention learning to obtain target thresholds for each modality. Using different thresholds for each modality, the module performs soft-thresholding to remove irrelevant noise and redundant features, focusing on the characteristics of the target in the different modal images. After fusion, a fused feature with more prominent target features is obtained. The multi-scale path aggregation module fully utilizes shallow features and, in combination with the position attention module, generates feature information that cascades shallow positional information and deep semantic information. The proposed scheme, utilizing infrared and visible light image features, can generate fused features that highlight target information in low-light nighttime scenes, improving target detection performance.

[0009] The technical solution specifically adopted by the present invention to solve the technical problem is:

[0010] A bimodal object detection model based on residual shrinkage attention network, including: a two-stream feature extraction network, a bimodal feature fusion module and a multi-scale path aggregation module;

[0011] The dual-stream feature extraction module includes a parallel dual-stream feature extraction network based on CSPDarknet, which takes the aligned infrared and visible light images as input data, extracts multi-scale features of the target in the dual-modal image, and outputs the corresponding feature map;

[0012] The bimodal feature fusion module is based on the residual shrinkage attention network, which performs adaptive soft thresholding denoising and enhancement on the extracted bimodal multi-scale feature maps. After fusion, a fusion feature with more significant target characteristics is obtained;

[0013] The multi-scale path aggregation module obtains multi-scale features that aggregate position information and semantic information by cascading four fusion features of different scales and combining them with the position attention module, and then feeds them into the YOLO detection module to obtain predicted target information.

[0014] Furthermore, the parallel two-stream feature extraction network based on CSPDarknet has two paths for extracting deep features of infrared and visible light images. Except for the different input images, the remaining extraction parts have the same structure and are composed of four CSPResn residual modules. Each CSPResn residual module is composed of three network minimum components CBS and n BottleNeck residual blocks.

[0015] Furthermore, the network's minimum component CBS is composed of a convolutional layer, a batch normalization BN layer, and a SiLU activation function connected in sequence; each of the BottleNeck residual blocks is composed of two CBS components and an Add tensor addition operation, and each CSPResn module is recorded as CSPRes1 or CSPRes3 according to the number of BottleNeck residual blocks it contains.

[0016] Furthermore, the dual-stream feature extraction module performs shallow feature extraction on small targets and deep feature extraction on large-scale targets, outputting feature maps of corresponding scales in each CSPResn. Depending on the category of the input image, the feature map resolutions are recorded from large to small as visible light image features V1, V2, V3, V4, and infrared image features I1, I2, I3, I4.

[0017] Furthermore, the dual-modal feature fusion module includes parallel residual shrinkage attention modules for different modalities, which are composed of residual attention, soft threshold function, channel addition operation, and residual connection operation. The images of different modalities are respectively learned by residual attention under their respective paths to obtain the threshold of the corresponding modality image target. The soft thresholding operation is performed using different thresholds to adaptively remove irrelevant noise and redundant features of the images of different modalities, and obtain the feature map of the focused target characteristics under each modality. The specific calculation formula is as follows:

[0018] X O =soft(RA(X I ),X I )+X I

[0019] Among them, X I represents the input features, X O Represents the output feature, RA(·) represents the residual attention, and soft(·) represents the soft threshold function. The residual attention RA consists of a global average pooling operation, two 1×1 convolutions, a RELU activation function, a Sigmoid function, a residual connection operation, and a channel multiplication operation. The specific calculation formula is as follows:

[0020] X'=GAP(X I )

[0021] X O =Sigmoid(f 1×1 (RELU(f 1×1 (X'))))·X'

[0022] Among them, GAP(·) represents the global average pooling operation, f 1×1(·) represents 1×1 convolution, RELU(·) represents RELU activation function, and Sigmoid(·) represents Sigmoid function.

[0023] The dual-modal feature fusion module adopts the residual attention mechanism to adaptively learn the thresholds of the targets in each modality image, and uses different thresholds for each modality to perform soft thresholding operations to remove irrelevant noise and redundant features, focusing on the characteristics of the targets in different modal images.

[0024] Furthermore, the four infrared image feature maps and visible light image feature maps of different scales extracted by the dual-stream feature extraction module are respectively input into the dual-modal feature fusion module for feature enhancement fusion to obtain four fused feature maps of different scales. The specific calculation formula is as follows:

[0025] F i =DMFFM(V i ,I i )

[0026] Among them, DMFFM(·) represents the dual-modal feature fusion module operation, V i Represents the visible light input characteristics, I i represents the infrared input feature, F i Indicates fusion features, i = 1, 2, 3, 4.

[0027] Furthermore, the multi-scale path aggregation module cascades feature maps of different scales, uses an additional layer of shallow features, and combines it with the CA attention module to consider the relationship between channels and long-distance position information. It fully utilizes the position information of shallow features and the semantic information of deep features to improve the model's ability to detect small targets. The specific working process includes the following steps:

[0028] Step S301: The obtained fusion feature map F i After 1×1 convolution, upsampling operation and fusion feature map F of the previous layer i-1 Perform Concat tensor splicing operation, CSPRes3 convolution, and obtain the intermediate feature map M of the semantic information of the cascaded aggregation deep features through the CA attention module i , where i = 1, 2, 3, 4;

[0029] Step S302: The intermediate feature map M i After downsampling operation and the intermediate feature map M of the next layer i+1 Perform Concat tensor splicing operation and CSPRes3 convolution to obtain the final feature map P of the position information of the cascaded aggregation shallow features j , where j = 1, 2, 3;

[0030] Step S3: Pass the feature map P1 to the detection layer YOLO1 to obtain the detection results of larger targets; then pass the feature map P2 to the detection layer YOLO2 to obtain the detection results of medium-sized targets; finally, pass the feature map P3 to the detection layer YOLO3 to obtain the detection results of small targets.

[0031] Furthermore, during the training process of the model, a one-to-one corresponding infrared and visible light image training set and validation set are used as input. After the training is completed, the optimal model is obtained and the model is tested using the test set to verify the effectiveness of the model.

[0032] The loss function introduces the vector angle between the true box and the predicted box of SIOU, and the constructed loss function Loss is as follows:

[0033]

[0034]

[0035] Among them, N is the number of detection layers, B i The number of targets whose labels are assigned to the priori boxes, S i ×S i is the number of grids into which the feature is divided, λ is the weight of the corresponding loss, λ1, λ2, and λ3 are set to 0.05, 0.3, and 0.7 respectively, and L SIOU is the positioning loss, Δ is the distance loss, Ω is the shape loss, L obj is the target confidence loss, L cls is the classification loss.

[0036] A bimodal object detection method based on residual shrinkage attention network includes the following steps:

[0037] Step S1: The aligned infrared and visible light images are used as input data and fed into a dual-stream feature extraction network to extract multi-scale features of the target in the dual-modal image and output the corresponding feature map;

[0038] Step S2: The extracted dual-modal multi-scale feature map is subjected to target feature adaptive soft threshold denoising and enhancement in the dual-modal feature fusion module. After fusion, a fusion feature with more significant target characteristics is obtained;

[0039] Step S3: In the multi-scale path aggregation module, the position attention module is combined to perform multi-scale path aggregation on the obtained bimodal fusion features, and the features are sent to the subsequent YOLO detection module for detection to obtain the predicted target information.

[0040] Compared with the existing technology, the present invention and its preferred embodiment propose a feature fusion module DMFFM based on the residual shrinkage attention network. It uses residual attention adaptive learning to obtain the threshold of the target in each modal image, uses different thresholds for each modality to perform soft thresholding operations, removes irrelevant noise and redundant features, focuses on the characteristics of the target in different modal images, and obtains more prominent fusion features after fusion. The multi-scale path aggregation module makes full use of shallow features, combines with the position attention module, and considers the relationship between channels and long-distance position information at the same time to obtain feature information of cascaded shallow position information and deep semantic information. Through the method of the present invention, more prominent fusion features of target information can be obtained in low-light scenes at night, improving the target detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0042] Figure 1 This is a diagram of the dual-modal target detection network structure in an embodiment of the present invention.

[0043] Figure 2 Schematic diagram of a feature fusion module based on a residual shrinkage attention network in an embodiment of the present invention.

[0044] Figure 3 Schematic diagram of the process of the dual-modal target detection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] To make the features and advantages of this patent more clearly understood, the following embodiments are specifically described in detail as follows:

[0046] It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by those skilled in the art to which this application belongs.

[0047] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0048] like Figure 3 As shown, this implementation first provides the design and training process of a bimodal object detection model based on a residual shrinkage attention network, which specifically includes the following steps:

[0049] Step 1: Obtain a dataset of infrared and visible light image alignments of different nighttime scenes and divide it into training, validation, and test sets.

[0050] Step 2: Build a parallel two-stream feature extraction network based on CSPDarknet, take the aligned infrared and visible light images as input, extract the deep features of the target, and output the corresponding multi-scale feature maps;

[0051] Step 3: Based on the residual shrinkage attention mechanism, a bimodal feature fusion module (DMMFFM) is designed to perform adaptive soft thresholding denoising and enhancement on the extracted bimodal multi-scale feature maps. After fusing them, a fusion feature with more significant target characteristics is obtained.

[0052] Step 4: Perform multi-scale path aggregation on the obtained bimodal fusion features, combine them with the position attention module, make full use of the position information of shallow features and the semantic information of deep features, and send them to the subsequent YOLO detection module for detection to obtain the predicted target information;

[0053] Step 5: Use the one-to-one corresponding infrared and visible light image training sets and validation sets as input to train the proposed dual-modal object detection network. During the training process, calculate the loss function, continuously backpropagate and update the model, select the model with the smallest loss and save it, and use the test set to test the model to test the effectiveness of the model.

[0054] Thus obtaining Figure 1 and Figure 2 The network model shown in Figure 3 is used to implement bimodal target detection based on residual shrinkage attention network.

[0055] Specifically, in step 1, the video frames containing one-to-one correspondence between infrared and visible light images were sampled every equal number of frames. Incompletely aligned infrared and visible light images were discarded, and well-aligned image pairs were retained. This yielded an aligned infrared and visible light dataset, which was then divided into training, test, and validation sets in a ratio of 7:2:1. Due to the low resolution of the infrared image acquisition device, the dataset was subjected to image contrast enhancement to improve the network's training and learning capabilities.

[0056] Specifically, in step 2, a parallel two-stream feature extraction network based on CSPDarknet extracts deep features from infrared and visible light images. Aside from the input images, the two paths share the same extraction structure. Based on a residual structure and incorporating the CSP concept, channel splitting reduces computational effort, maintaining good performance while building a deep residual network. It primarily consists of four CSPResn residual modules. Each CSPResn residual module is composed of three CBS (Convolutional Broadsets) components and n BottleNeck residual blocks. The CBS component consists of a convolutional layer, a batch normalization (BN) layer, and a SiLU activation function, all connected in sequence. Each BottleNeck residual block consists of two CBS components and an Add tensor addition operation. Each CSPResn module can be denoted as CSPRes1, CSPRes3, CSPRes3, and CSPRes1, depending on the number of BottleNeck residual blocks it contains.

[0057] At the same time, as a preference, in step 2, the dual-stream feature extraction network will perform shallow feature extraction on small targets and deep feature extraction on large-scale targets, and output feature maps of corresponding scales in each CSPResn. Depending on the category of the input image, the feature map resolutions are recorded from large to small as visible light image features V1, V2, V3, V4, and infrared image features I1, I2, I3, I4.

[0058] Specifically, in step 3, the dual-modal feature fusion module based on the residual shrinkage attention network includes parallel residual shrinkage attention modules of different modalities. The residual shrinkage attention module consists of a residual attention, a soft threshold function, a channel addition operation, and a residual connection operation. Images of different modalities are respectively obtained by residual attention learning under their respective paths to obtain the threshold of the corresponding modality image target, and soft thresholding operations are performed using different thresholds to adaptively remove irrelevant noise and redundant features of images of different modalities, and obtain feature maps of the focused target characteristics under each modality. The specific calculation formula is as follows:

[0059] X O =soft(RA(X I ),X I )+X I

[0060] Among them, X I represents the input features, X O Represents the output feature, RA(·) represents the residual attention, and soft(·) represents the soft threshold function. The residual attention RA consists of a global average pooling operation, two 1×1 convolutions, a RELU activation function, a Sigmoid function, a residual connection operation, and a channel multiplication operation. The specific calculation formula is as follows:

[0061] X'=GAP(X I )

[0062] X O =Sigmoid(f 1×1 (RELU(f 1×1 (X'))))·X'

[0063] Among them, GAP(·) represents the global average pooling operation, f 1×1 (·) represents 1×1 convolution, RELU(·) represents RELU activation function, and Sigmoid(·) represents Sigmoid function.

[0064] Specifically, in step 3, the infrared image feature maps and visible light image feature maps of four different scales are extracted and respectively input into the dual-modal feature fusion module DMFFM for feature enhancement fusion to obtain four fused feature maps of different scales. The specific calculation formula is as follows:

[0065] F i =DMFFM(V i ,I i )

[0066] Among them, DMFFM(·) represents the dual-modal feature fusion module operation, V i Represents the visible light input characteristics, I i represents the infrared input feature, F i Indicates fusion features.

[0067] Specifically, in step 4, the obtained fusion feature map is subjected to multi-scale path aggregation, an additional layer of shallow features is used, and the position attention module is combined to fully utilize the position information of the shallow features and the semantic information of the deep features. The map is input into the subsequent YOLO detection module for target detection tasks to obtain the final detection result, including the following steps:

[0068] The obtained fusion feature map F i After 1×1 convolution, upsampling operation and fusion feature map F of the previous layer i-1 Perform Concat tensor splicing operation, CSPRes3 convolution, and obtain the intermediate feature map M of the semantic information of the cascaded aggregation deep features through the CA attention module i , where i = 1, 2, 3, 4;

[0069] The intermediate feature map M i After downsampling operation and the intermediate feature map M of the next layer i+1 Perform Concat tensor splicing operation and CSPRes3 convolution to obtain the final feature map P of the position information of the cascaded aggregation shallow features j, where j = 1, 2, 3;

[0070] The feature map P1 is passed to the detection layer YOLO1 to obtain the detection results of larger targets; the feature map P2 is passed to the detection layer YOLO2 to obtain the detection results of medium-scale targets; the feature map P3 is passed to the detection layer YOLO3 to obtain the detection results of small targets.

[0071] Specifically, in step 5, the loss function of the bimodal object detection network introduces the vector angle between the SIOU real box and the predicted box, which helps to converge the training and improve the effect. The constructed loss function Loss is as follows:

[0072]

[0073]

[0074] Among them, N is the number of detection layers, B i The number of targets whose labels are assigned to the priori boxes, S i ×S i is the number of grids that the feature is divided into. λ is the weight of the corresponding loss, and λ1, λ2, and λ3 are set to 0.05, 0.3, and 0.7 respectively. SIOU is the positioning loss, Δ is the distance loss, and Ω is the shape loss. obj is the target confidence loss, L cls is the classification loss.

[0075] The model provided by the present invention is further introduced below:

[0076] See also Figure 1The following diagram shows the architecture of a dual-modal object detection network. The network uses a parallel two-stream feature extraction network based on CSPDarknet to extract deep features from infrared and visible light images. Aside from the input images, the two paths share the same extraction architecture. Based on a residual architecture and incorporating the CSP concept, channel splitting reduces computational effort, enabling the construction of a deep residual network while maintaining good performance. The network primarily consists of four CSPResn residual modules. Each CSPResn residual module is composed of three CBS (Convolutional Broadsets) components and n BottleNeck residual blocks. The CBS component consists of a convolutional layer, a batch normalization (BN) layer, and a SiLU activation function. Each BottleNeck residual block consists of two CBS components and an Add tensor operation. Each CSPResn module can be denoted as CSPRes1, CSPRes3, CSPRes3, and CSPRes1, depending on the number of BottleNeck residual blocks it contains. Using different numbers of BottleNeck residual blocks allows for deeper feature extraction networks, thereby extracting deeper image features.

[0077] Among them, for the input infrared and visible light images, the two-stream feature extraction network will perform shallow feature extraction for small targets and deep feature extraction for large-scale targets, and output feature maps of corresponding scales in each CSPResn. According to the different categories of the input images, the feature map resolutions are recorded from large to small as visible light image features V1, V2, V3, V4, and infrared image features I1, I2, I3, I4.

[0078] In this embodiment, as a preferred solution, a feature extraction network inputs a visible light image with a length, width, and number of channels of 640×640×3, which is adjusted to a feature map of 320×320×16 through a CBS with a convolution size of 6×6. Feature extraction is then performed in CSPRes1, CSPRes3, CSPRes3, and CSPRes1, and feature maps of sizes of 160×160×32, 80×80×64, 40×40×128, and 20×20×256 are output, respectively, to extract visible light feature maps V1, V2, V3, and V4; similarly, another parallel feature extraction network obtains infrared feature maps I1, I2, I3, and I4 of sizes of 160×160×32, 80×80×64, 40×40×128, and 20×20×256 through the same steps.

[0079] In this embodiment, the dual-modal feature fusion module based on the residual shrinkage attention network includes parallel residual shrinkage attention modules of different modalities. The residual shrinkage attention module consists of a residual attention, a soft threshold function, a channel addition operation, and a residual connection operation. Images of different modalities are respectively obtained by residual attention learning under their respective paths to obtain the threshold of the corresponding modality image target, and different thresholds are used for soft thresholding operations to adaptively remove irrelevant noise and redundant features of images of different modalities, and obtain feature maps of the focused target characteristics under each modality. The specific calculation formula is as follows:

[0080] X O =soft(RA(X I ),X I )+X I

[0081] Among them, X I represents the input features, X O Represents the output feature, RA(·) represents the residual attention, and soft(·) represents the soft threshold function. The residual attention RA consists of a global average pooling operation, two 1×1 convolutions, a RELU activation function, a Sigmoid function, a residual connection operation, and a channel multiplication operation. The specific calculation formula is as follows:

[0082] X'=GAP(X I )

[0083] X O =Sigmoid(f 1×1 (RELU(f 1×1 (X'))))·X'

[0084] Among them, GAP(·) represents the global average pooling operation, f 1×1 (·) represents 1×1 convolution, RELU(·) represents RELU activation function, and Sigmoid(·) represents Sigmoid function.

[0085] Among them, see Figure 2 The extracted infrared image feature maps and visible light image feature maps of four different scales are input into the dual-modal feature fusion module DMFFM for feature enhancement and fusion, and four fused feature maps of different scales are obtained. The specific calculation formula is as follows:

[0086] F i =DMFFM(V i ,I i )

[0087] Among them, DMFFm(·) represents the dual-modal feature fusion module operation, V i Represents the visible light input characteristics, I irepresents the infrared input feature, F i Indicates fusion features.

[0088] In this embodiment, the input visible light feature map V i and infrared characteristic map I i First, global average pooling (GAP) is used to extract global feature information; two 1×1 convolutions are used to reduce the number of calculated parameters; a Sigmoid function outputs the feature information to (0,1) to increase the nonlinearity of the network and thus capture the nonlinear relationship between the data; finally, a residual connection is used to avoid gradient disappearance; through residual attention, different thresholds for each data feature are adaptively learned when processing different data features, and then soft thresholding is used to remove features unrelated to the target features and enhance the expressiveness of the features. The resulting fused feature maps of sizes 160×160×32, 80×80×64, 40×40×128, and 20×20×256 are denoted as F1, F2, F3, and F4.

[0089] Finally, the obtained fusion feature map is subjected to multi-scale path aggregation, and an additional layer of shallow features is used. Combined with the position attention module, while considering the relationship between channels and long-distance position information, the position information of the shallow features and the semantic information of the deep features are fully utilized. The map is input into the subsequent YOLO detection module for target detection tasks to obtain the detection results. The specific steps are as follows:

[0090] The obtained fusion feature map F i After 1×1 convolution, upsampling operation and fusion feature map F of the previous layer i-1 Perform Concat tensor splicing operation, CSPRes3 convolution, and obtain the intermediate feature map M of the semantic information of the cascaded aggregation deep features through the CA attention module i , where i = 1, 2, 3, 4;

[0091] The intermediate feature map M i After downsampling operation and the intermediate feature map M of the next layer i+1 Perform Concat tensor splicing operation and CSPRes3 convolution to obtain the final feature map P of the position information of the cascaded aggregation shallow features j , where j = 1, 2, 3;

[0092] The feature map P1 is passed to the detection layer YOLO1 to obtain the detection results of larger targets; the feature map P2 is passed to the detection layer YOLO2 to obtain the detection results of medium-scale targets; the feature map P3 is passed to the detection layer YOLO3 to obtain the detection results of small targets.

[0093] The multi-scale path aggregation module incorporates top-down paths and horizontally connected feature pyramid networks (FPNs) to fuse high- and low-level features. Combined with the CA attention module, it considers inter-channel relationships and long-distance positional information. It also leverages the high resolution of low-level features and the high semantic information of high-level features to enhance the semantic information of different features. To promote the flow of information, bottom-up path aggregation is combined to allow high-level features to benefit from the rich positional information provided by the lower layers. At the same time, a shallow auxiliary network is added, and the shallow feature maps can obtain more positional information of small objects, improving the model's ability to detect small objects.

[0094] In this embodiment, as a preference, after the fusion feature maps F1, F2, F3, and F4 pass through the multi-scale path aggregation module, the sizes of the features to be detected are 80×80×64, 40×40×128, and 20×20×256, respectively, denoted as P1, P2, and P3. Taking the small feature map P3 as an example, it is passed to the YOLO3 detection layer, and a convolution with a convolution dimension of (4+1+2)×3 is used for target prediction, where 4 represents the regression parameter for determining each feature point. After the regression parameter is adjusted, a prediction box can be obtained, 1 represents whether each feature point contains an object, 2 represents the type of object contained in each feature point, and 3 represents the default preset of 3 verification boxes. After the convolution outputs a feature map of size (1×20×20×21), the network's prediction result for large-scale targets can be obtained after decoding.

[0095] In order to better illustrate the effectiveness of the present invention, the embodiment of the present invention also uses a comparative experiment to compare the reconstruction effects.

[0096] Specifically, an embodiment of the present invention uses the infrared and visible light target detection dataset KAIST. Due to the misalignment problem and labeling errors of infrared and visible light images in the original training data, only completely aligned image pairs are retained, including 2272 pairs of infrared and visible image pairs, covering both daytime and nighttime low-light scenes, and two categories: "pedestrians" and "people".

[0097] The network proposed in this invention uses the pytorch deep learning framework, the experimental platform is Ubuntu 18.04, the CUDA version is 10.1, the graphics card is NVIDIA Quard P2000, and the video memory is 5G. The model training input image size is set to 640×640, the training epochs is 300, the training loss is normalized, and the label smoothing is set to 0.01. The stochastic gradient descent (SGD) optimizer is used, with an initial learning rate of 0.01, a momentum of 0.937, a weight decay of 0.0005, and a minimum learning rate of 0.0001. At the same time, the frozen training method is used to speed up the model training. The epochs of frozen training are set to 50, the batch-size of frozen training is set to 8, and the batch-size after thawing is set to 4.

[0098] The model performance is evaluated using various mean average precision (mAP) indicators of target detection, and the IOU threshold is set to 50%.

[0099] The comparative experiment selected five representative target detection methods to compare with the experimental results of this invention, including four single-modal target detection methods and one dual-modal target detection method. The experimental results are shown in Table 1. The five representative target detection methods include:

[0100] YOLOv3: From the reference "Redmon J, Farhadi A. YOLOv3: An Incremental Improvement [J]. arXiv e-prints, 2018".

[0101] YOLOv4-tiny: From reference “https: / / github.com / AlexeyAB / darknet”.

[0102] YOLOv5: From reference “https: / / github.com / ultralytics / yolov5”.

[0103] YOLOv7: From the reference "Wang CY, BochkovskiyA, Liao HY M.YOLOv7: Trainablebag-of-freebies sets new state-of-the-art for real-time object detectors[J].arXiv preprint arXiv:2207.02696,2022"

[0104] CFT: From the reference "Qingyun F, Dapeng H, Zhaokui W. Cross-Modality Fusion Transformer for Multispectral Object Detection[J].arXiv e-prints, 2021".

[0105] Table 1 Comparative experiments of different methods

[0106] Method Data mAP / % mono-modality networks YOLOv3 RGB 76.71 YOLOv3 IR 81.97 YOLOv4-tiny RGB 70.17 YOLOv4-tiny IR 80.13 YOLOv5 RGB 86.18 YOLOv5 IR 93.10 YOLOv7 RGB 91.23 YOLOv7 IR 94.27 multi-modality networks CFT RGB+IR 94.30 Baseline RGB+IR 93.37 DMPDNet RGB+IR 96.25

[0107] It can be found from Table 1 that the dual-modal target detection method proposed in this embodiment is superior to the comparison algorithm in terms of accuracy.

[0108] In summary, it can be seen that the present invention addresses the problem of poor detection accuracy of a single-modal target detection algorithm caused by unclear target features under low illumination conditions at night. A dual-modal target detection network model using infrared and visible light images is constructed, and a dual-modal feature fusion module based on a residual shrinkage attention network is adopted. The target thresholds of different modal features are obtained through residual attention adaptive learning, and the noise and redundant information in the image features are removed by a soft threshold function, focusing on the target features, so that the network can obtain complementary detail features of infrared and visible light, and better fuse target feature information; through a multi-scale path aggregation module, a four-layer feature map is used, combined with a position attention module, the position information of shallow features and the semantic information of deep features are fully utilized, effectively improving the model's target detection effect for complex scenes at night.

[0109] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0110] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0111] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0113] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

[0114] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive various other forms of dual-modal target detection models and methods based on residual shrinkage attention networks under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of this invention should be covered by this patent.

Claims

1. A bimodal target detection model based on residual shrinkage attention network, characterized by: include: Dual-stream feature extraction module, dual-modal feature fusion module and multi-scale path aggregation module; The dual-stream feature extraction module includes a parallel dual-stream feature extraction network based on CSPDarknet, which takes the aligned infrared and visible light images as input data, extracts multi-scale features of the target in the dual-modal image, and outputs the corresponding feature map; The bimodal feature fusion module is based on the residual shrinkage attention network, which performs adaptive soft thresholding denoising and enhancement on the extracted bimodal multi-scale feature maps. After fusion, a fusion feature with more significant target characteristics is obtained; The dual-modal feature fusion module includes parallel residual shrinkage attention modules of different modalities, which are composed of residual attention, soft threshold function, channel addition operation, and residual connection operation. The images of different modalities are respectively learned by residual attention under their respective paths to obtain the threshold of the corresponding modality image target. The soft thresholding operation is performed using different thresholds to adaptively remove irrelevant noise and redundant features of the images of different modalities, and obtain the feature map of the focused target characteristics under each modality. The multi-scale path aggregation module cascades four fusion features of different scales, uses an additional layer of shallow features, and combines it with the position attention module. It also considers the relationship between channels and long-distance position information to fully utilize the position information of shallow features and the semantic information of deep features, thereby improving the model's ability to detect small targets. It obtains multi-scale features that aggregate position information and semantic information, and sends them to the YOLO detection module to obtain predicted target information. The adopted loss function introduces the vector angle between the true box and the predicted box of SIOU.

2. The dual-modal object detection model based on residual shrinkage attention network according to claim 1, characterized in that: The parallel two-stream feature extraction network based on CSPDarknet extracts deep features of infrared and visible light images. Except for the input images, the two paths have the same extraction structure. They are composed of four CSPResn residual modules. Each CSPResn residual module is composed of three network minimum components (CBS) and n BottleNeck residual blocks.

3. The dual-modal object detection model based on residual shrinkage attention network according to claim 2, characterized in that: The network's smallest component, CBS, is composed of a convolutional layer, a batch normalization (BN) layer, and a SiLU activation function connected in sequence. Each BottleNeck residual block consists of two CBS components and an Add tensor addition operation. Each CSPResn module is recorded as CSPRes1 or CSPRes3 according to the number of BottleNeck residual blocks it contains.

4. The dual-modal object detection model based on residual shrinkage attention network according to claim 3, characterized in that: The dual-stream feature extraction module performs shallow feature extraction on small targets and deep feature extraction on large-scale targets, outputting feature maps of corresponding scales in each CSPResn. Depending on the category of the input image, the feature map resolutions are recorded from large to small as visible light image features V1, V2, V3, V4, and infrared image features I1, I2, I3, I4.

5. The dual-modal object detection model based on residual shrinkage attention network according to claim 1, characterized in that: The specific calculation formula of the dual-modal feature fusion module is as follows: X O =soft(RA(X I ),X I )+X I Among them, X I represents the input features, X O Represents the output feature, RA(·) represents the residual attention, and soft(·) represents the soft threshold function. The residual attention RA consists of a global average pooling operation, two 1×1 convolutions, a RELU activation function, a Sigmoid function, a residual connection operation, and a channel multiplication operation. The specific calculation formula is as follows: X'=GAP(X I ) X O =Sigmoid(f 1×1 (RELU(f 1×1 (X'))))·X' Among them, GAP(·) represents the global average pooling operation, f 1×1 (·) represents 1×1 convolution, RELU(·) represents RELU activation function, and Sigmoid(·) represents Sigmoid function.

6. The dual-modal object detection model based on residual shrinkage attention network according to claim 5, characterized in that: The four infrared image feature maps and visible light image feature maps of different scales extracted by the dual-stream feature extraction module are respectively input into the dual-modal feature fusion module for feature enhancement and fusion to obtain four fused feature maps of different scales. The specific calculation formula is as follows: F i =DMFFM(V i ,I i ) Among them, DMFFM(·) represents the dual-modal feature fusion module operation, V i Represents the visible light input characteristics, I i represents the infrared input feature, F i Indicates fusion features, i = 1, 2, 3, 4.

7. The dual-modal object detection model based on residual shrinkage attention network according to claim 1, characterized in that: The specific working process of the multi-scale path aggregation module includes the following steps: Step S301: The obtained fusion feature map F i After 1×1 convolution, upsampling operation and fusion feature map F of the previous layer i-1 Perform Concat tensor splicing operation, CSPRes3 convolution, and obtain the intermediate feature map M of the semantic information of the cascaded aggregation deep features through the CA module i , where i = 1, 2, 3, 4; Step S302: The intermediate feature map M i After downsampling operation and the intermediate feature map M of the next layer i+1 Perform Concat tensor splicing operation and CSPRes3 convolution to obtain the final feature map P of the position information of the cascaded aggregation shallow features j , where j = 1, 2, 3; Step S303: Pass the feature map P1 to the detection layer YOLO1 to obtain the detection results of larger targets; then pass the feature map P2 to the detection layer YOLO2 to obtain the detection results of medium-sized targets; finally, pass the feature map P3 to the detection layer YOLO3 to obtain the detection results of small targets.

8. The dual-modal object detection model based on residual shrinkage attention network according to claim 1, characterized in that: During the training process of the model, a one-to-one corresponding infrared and visible light image training set and validation set are used as input. After the training is completed, the best model is obtained and the test set is used to test the model to verify its effectiveness. The constructed loss function Loss is as follows: Among them, N is the number of detection layers, B i The number of targets whose labels are assigned to the priori boxes, S i ×S i is the number of grids into which the feature is divided, λ is the weight of the corresponding loss, λ1, λ2, and λ3 are set to 0.05, 0.3, and 0.7 respectively, and L SIOU is the positioning loss, Δ is the distance loss, Ω is the shape loss, L obj is the target confidence loss, L cls is the classification loss.

9. A method for dual-modal target detection based on a residual shrinkage attention network, based on the dual-modal target detection model based on a residual shrinkage attention network according to any one of claims 1 to 8, characterized in that: The following steps are involved: Step S1: The aligned infrared and visible light images are used as input data and fed into a dual-stream feature extraction network to extract multi-scale features of the target in the dual-modal image and output the corresponding feature map; Step S2: The extracted dual-modal multi-scale feature map is subjected to target feature adaptive soft threshold denoising and enhancement in the dual-modal feature fusion module. After fusion, a fusion feature with more significant target characteristics is obtained; Step S3: In the multi-scale path aggregation module, the position attention is combined to perform multi-scale path aggregation on the obtained bimodal fusion features, and the features are sent to the subsequent YOLO detection module for detection to obtain the predicted target information.

Images