Visible light-thermal imaging target detection method based on double-flow network fusion
Through the visible-thermal imaging object detection method based on dual-stream network fusion, the feature fusion of RGB and IR images is used to solve the insufficient detection of RGB images in low-light environments, and the accuracy and robustness of object detection are improved.
Patent Information
- Application Number
- CN202510325760.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-19
- Publication Date
- 2025-08-01
AI Technical Summary
In the prior art, RGB images have insufficient detection accuracy and robustness in low light and bad weather environments, and infrared images lack rich color and texture information, making it difficult to effectively integrate the two modal information to improve the accuracy and robustness of object detection.
The visible-thermal imaging object detection method based on dual-stream network fusion is adopted. By constructing the paired RGB and IR images into four channels of input data, RGB and IR features are extracted respectively, and feature fusion is used by multi-head attention modules, combining multi-layer perceptron processing and backpropagation algorithm training model, the accuracy and robustness of object detection are enhanced.
It improves the accuracy and robustness of object detection, can effectively identify targets in complex environments, make up for the missing information of RGB images, and enhances the description of spatial position information.
Smart Images
Figure CN120411697A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision and pattern recognition, and in particular to a visible light-thermal imaging target detection method based on dual-stream network fusion. Background Art
[0002] With the rapid development of deep learning and computer vision technologies, object detection has found widespread application in areas such as autonomous driving, security surveillance, and military reconnaissance. Traditional object detection algorithms primarily rely on single RGB images, but the quality and reliability of RGB images can significantly degrade in complex environments such as low light and inclement weather. Infrared (IR) images offer unique advantages in nighttime and low-visibility environments, but they lack rich color and texture information.
[0003] To fully utilize the complementarity of RGB and IR images, multimodal fusion methods have emerged in the existing technology. However, how to effectively fuse the information of the two modalities to improve the accuracy and robustness of object detection remains a challenging problem. Summary of the Invention
[0004] The present invention provides a visible light-thermal imaging target detection method based on dual-stream network fusion, which can solve the problem of insufficient target detection in complex scenes in the prior art.
[0005] The present invention provides a visible light-thermal imaging target detection method based on dual-stream network fusion, wherein the method comprises:
[0006] The paired RGB images and corresponding IR images in the dataset are constructed into four-channel input data, where the RGB image is three channels and the IR image is a single channel;
[0007] Normalize the input data to a predetermined size and pass the normalized input data to a two-stream fusion feature extraction neural network to extract RGB features and IR features respectively;
[0008] The RGB feature extractor of the dual-stream fusion feature extraction neural network extracts features from the visible light image to obtain the RGB feature map F rgb ;
[0009] The IR feature extractor of the dual-stream fusion feature extraction neural network extracts the spatial information of the infrared image to obtain the IR feature map F ir ;
[0010] The RGB feature map F rgb and IR feature map F ir After flattening, parallel splicing is performed to obtain the spliced feature vector I fu ;
[0011] Spliced Feature Vector I fu It is successively processed by 3 multi-layer perceptrons to generate the corresponding query matrix Q, key matrix K, and value matrix V, and the generated query matrix Q, key matrix K, and value matrix V are input into the multi-head attention module to calculate the interaction feature vector I after feature fusion mid ;
[0012] Interaction Feature Vector I after Feature Fusion mid After being normalized and processed by a multi-layer perceptron, it is used as a residual and spliced together with the interaction feature vector I after feature fusion mid to form a new spliced feature vector I' fu ;
[0013] The new spliced feature vector I' is subjected to object detection through a detection head fu ;
[0014] The backpropagation algorithm and stochastic gradient descent method are used to reduce the overall prediction error L all to train the object detection model, and the final RGB-IR feature fusion object detection model is obtained after multiple iterations of training
[0015] The final RGB-IR feature fusion object detection model is used for object detection on an open-source dataset
[0016] Preferably, the spliced feature vector I fu is:
[0017] I fu = f Concat (f flatten (F rgb ), f flatten (F ir ))
[0018] where f Concat is a parallel splicing function, f flatten (F rgb ) is the flattened RGB feature map, and f flatten (F ir ) is the flattened IR feature map
[0019] Preferably, the query matrix Q, key matrix K, and value matrix V are respectively:
[0020] Q = MLP Q (I fu )
[0021] K = MLP K (I fu )
[0022] V = MLP V (Ifu )
[0023] Among them, MLP is a multi - layer perceptron module, and MLP Q , MLP K , MLP V respectively represent 3 non - reused multi - layer perceptrons.
[0024] Preferably, the interactive feature vector I after feature fusion mid is:
[0025] I mid = MHA(Q, K, V),
[0026] where MHA is a multi - head attention module.
[0027] Preferably, the predetermined size is 640×640×4 pixels.
[0028] Preferably, the Resnet module of the RGB feature extractor includes a MaxPool layer, a 1x1Conv layer, and a 3x3Conv layer connected in sequence.
[0029] Preferably, the Resnet module of the IR feature extractor includes a MaxPool layer and two consecutive 1x1Conv layers.
[0030] Through the above technical solutions, the paired RGB and IR images can be first constructed into four - channel input data and fed into the deep convolutional neural network model. By separately extracting RGB and IR features and using the attention mechanism for feature fusion. Then, using the multi - head attention module to focus on and fuse the features of each modality can not only capture the detailed features of the target but also enhance the description of the spatial position information, improving the accuracy and robustness of target detection. Description of the Drawings
[0031] The accompanying drawings included are used to provide a further understanding of the embodiments of the present invention, which form a part of the specification, are used to illustrate the embodiments of the present invention, and together with the written description are used to explain the principles of the present invention. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.
[0032] Figure 1 Shows a flowchart of a visible - light - thermal imaging target detection method based on dual - stream network fusion according to an embodiment of the present invention;
[0033] Figure 2 Shows a schematic diagram of the construction of four - channel input data according to an embodiment of the present invention;
[0034] Figure 3 Shows a schematic diagram of a dual-stream network fusion model according to an embodiment of the present invention. Detailed implementation manners
[0035] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other. The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and in no way limits the present invention and its application or use. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.
[0036] It should be noted that the terms used herein are only for describing specific implementation manners and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0037] Unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present invention. At the same time, it should be understood that, for the sake of convenience of description, the dimensions of the various parts shown in the drawings are not drawn in actual proportional relationships. Technologies, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and devices should be regarded as part of the authorized specification. In all the examples shown and discussed herein, any specific value should be construed as merely exemplary and not as a limitation. Therefore, other examples of the exemplary embodiments may have different values. It should be noted that: like reference numerals and letters denote like items in the following drawings, and thus, once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.
[0038] Figure 1 Shows a flowchart of a visible light-thermal imaging target detection method based on dual-stream network fusion according to an embodiment of the present invention.
[0039] As Figure 1 shown, the embodiments of the present invention provide a visible light (RGB)-thermal imaging target detection method based on dual-stream network fusion, wherein the method includes:
[0040] Construct the paired RGB images and corresponding IR images in the dataset into four-channel input data (as shown in Figure 2 ), where a in Figure 2 is the RGB image, b in Figure 2 is the IR image, and c in Figure 2 is the newly constructed four-channel input). Among them, the RGB image has three channels and the IR image has a single channel;
[0041] Normalize the input data to a predetermined size and transmit the normalized input data to a two-stream fusion feature extraction neural network (for example, a deep convolutional neural network model) to extract RGB features and IR features respectively;
[0042] Specifically, the two-stream fusion feature extraction neural network includes two feature extraction streams, which process the RGB and IR channels respectively. The network contains multiple convolutional layers and fully connected layers for extracting features of the two modalities, as shown in Figure 3 . Among them, a in Figure 3 is the system architecture, and b in Figure 3 is the dyfusion module.
[0043] The RGB feature extractor of the two-stream fusion feature extraction neural network extracts the features in the visible light image to obtain the RGB feature map F rgb ;
[0044] Specifically, the RGB feature extractor is used to extract rich detail features in the visible light image to obtain the RGB feature map F rgb to make up for the lack of information in the IR image.
[0045] The IR feature extractor of the two-stream fusion feature extraction neural network extracts the spatial information of the infrared image to obtain the IR feature map F ir ;
[0046] Specifically, the IR feature extractor aims to extract the spatial information of the infrared image to obtain the IR feature map F ir to make up for the lack of information in the RGB image in a complex environment.
[0047] Flatten the RGB feature map F rgb and the IR feature map F ir and then concatenate them in parallel to obtain the concatenated feature vector I fu ;
[0048] The concatenated feature vector I fuIt is successively processed by 3 multi-layer perceptrons to generate the corresponding query matrix Q, key matrix K, and value matrix V, and the generated query matrix Q, key matrix K, and value matrix V are input into the multi-head attention module to calculate the interactive feature vector I after feature fusion mid ;
[0049] The interactive feature vector I after feature fusion mid After being normalized and processed by a multi-layer perceptron, it is used as a residual and spliced with the interactive feature vector I after feature fusion mid together to form a new spliced feature vector I’ fu ;
[0050] The new spliced feature vector I’ is subjected to object detection through a detection head (for example, the detection head of yolov7) fu for object detection;
[0051] The backpropagation algorithm and the stochastic gradient descent method are adopted to reduce the overall prediction error L all to train the object detection model, and the final RGB-IR feature fusion object detection model is obtained after multiple iterations of training;
[0052] The final RGB-IR feature fusion object detection model is used for object detection on an open-source dataset
[0053] For example, first, the RGB image and the corresponding IR image are constructed into four-channel input data, normalized to a unified size (such as 640×640×4 pixels), and fed into the trained model to detect the test results
[0054] Through the above technical solutions, the paired RGB and IR images can be first constructed into four-channel input data and fed into the deep convolutional neural network model. By separately extracting the RGB and IR features and using the attention mechanism for feature fusion. Then, the multi-head attention module is used to focus on and fuse the features of each modality, which can not only capture the detailed features of the target but also enhance the description of the spatial position information, improving the accuracy and robustness of object detection
[0055] Among them, by constructing the RGB image and the corresponding IR image into four-channel input data and feeding them into the two-stream fusion network for feature extraction: the IR image can provide the spatial position information of the target, making up for the deficiencies of the RGB image in low-light or complex environments, while the RGB image can provide rich texture and detail information, and the combination of the two improves the accuracy and robustness of object detection
[0056] Furthermore, the feature extraction module of the two-stream network generates feature maps for the RGB and IR images respectively, and uses the attention mechanism to calculate the attention weights of the features of each modality. The multi-head attention module fuses the RGB and IR features to generate a fused feature vector that captures the detailed features of the target while highlighting the target's spatial information and effectively suppressing background noise.
[0057] According to one embodiment of the present invention, the multimodal feature fusion mechanism can use a regional-level triplet constraint (i.e., the target detection model uses a triplet loss function to constrain features, shortening the distance between the fused features and the target features while increasing the distance from the background features) to optimize the RGB features, IR features, and fused features respectively. Thus, through this constraint mechanism, the global features and the features from the target region can be brought close in the feature space while being away from the background region features, thereby enhancing the robustness and accuracy of target detection for complex scenes.
[0058] According to one embodiment of the present invention, the concatenated feature vector I fu for:
[0059] I fu =f Concat (f flatten (F rgb ),f flatten (F ir )),
[0060] Among them, f Concat is the parallel splicing function, f flatten (F rgb ) is the flattened RGB feature map, f flatten (F ir ) is the flattened IR feature map.
[0061] According to one embodiment of the present invention, the query matrix Q, the key matrix K, and the value matrix V are respectively:
[0062] Q=MLP Q (I fu ),
[0063] K=MLP K (I fu ),
[0064] V=MLP V (I fu ),
[0065] Among them, MLP is a multi-layer perceptron module. Q 、MLP K 、MLP V They represent three non-reused multi-layer perceptrons.
[0066] According to an embodiment of the present invention, the interactive feature vector I after feature fusion mid is as follows:
[0067] I mid = MHA(Q, K, V),
[0068] where MHA is the multi-head attention module.
[0069] According to an embodiment of the present invention, the predetermined size is 640×640×4 pixels.
[0070] According to an embodiment of the present invention, the Resnet module of the RGB feature extractor includes a MaxPool layer, a 1x1Conv layer, and a 3x3Conv layer connected in sequence.
[0071] Specifically, for the Resnet module in the RGB feature extractor, the MaxPool layer is connected first, followed by the 1x1Conv layer and the 3x3Conv layer. The 3x3Conv layer is used to increase the receptive field and extract detailed feature information.
[0072] According to an embodiment of the present invention, the Resnet module of the IR feature extractor includes a MaxPool layer and two consecutive 1x1Conv layers.
[0073] Specifically, the MaxPool layer can reduce the useless background information in the feature map, and the two consecutive 1x1Conv layers can linearly combine multiple feature maps to achieve cross-channel information integration.
[0074] Next, a visible light-thermal imaging target detection method based on dual-stream network fusion according to the present invention will be described with examples.
[0075] The visible light-thermal imaging image target recognition method based on dual-stream network fusion of the present invention introduces a dual-stream network to extract rich texture and information details of RGB images, and IR images provide the spatial position and contour information of the target, and uses a multi-head attention mechanism to weight and fuse the feature values of the two modalities to improve the target recognition accuracy. Taking a large visible light-thermal imaging database as an example, it contains 4129 pairs of paired training samples and 1013 pairs of paired test samples.
[0076] Figure 1 is the flowchart of the dual-stream network fusion method of the present invention. As Figure 1 [[ID=,39]]shown, the present invention specifically includes the following steps:
[0077] Step S0, construct the 4129 pairs of RGB images and IR images in the dataset into four-channel input data, as Figure 2As shown, 4129 input data are obtained in this way;
[0078] Step S1: Normalize the input data in S0 to a unified size (such as 640×640×4 pixels), and send it into a two-stream fusion feature extraction neural network, which consists of two feature extraction streams (processing the RGB and IR channels respectively), including multiple convolutional layers and fully connected layers, and is used to extract the features of the two modalities, as Figure 3 shown;
[0079] Step S2: The RGB feature extractor is used to extract the rich detail features in the visible light image to obtain the RGB feature map F rgb to make up for the lack of information in the IR image. In the RGB feature extractor, the Resnet module is first connected to the MaxPool layer, then to the 1x1Conv layer and the 3x3Conv layer. The 3x3Conv layer is used to increase the receptive field and extract the detailed feature information;
[0080] Step S3: The IR feature extractor aims to extract the spatial information of the infrared image to obtain the IR feature map F ir , to make up for the lack of information in the RGB image in complex environments. The Resnet module of the IR feature extractor consists of the MaxPool layer and two consecutive 1x1Conv layers. The MaxPool layer can reduce the useless background information in the feature map, and the two consecutive 1x1Conv layers can linearly combine multiple feature maps to achieve cross-channel information integration;
[0081] Step S4: Use the F rgb and F ir feature maps obtained in S2 and S3, flatten them and concatenate them in parallel to obtain the concatenated feature vector I fu : I fu = f Concat (f flatten (F[[ID=�1]] rgb ), f flatten (F ir ));
[0082] Step S5: The concatenated feature vector I fu obtained in S4 is successively processed by 3 multi-layer perceptrons to generate the corresponding Q, K, V matrices: Q = MLP Q (I fu ), K = MLP K (I fu ), V = MLP V (I fu ), where MLP is the multi-layer perceptron module, and then input into the multi-head attention module to calculate the interactive feature vector I mid : I mid= MHA(Q, K, V), where MHA is the multi-head attention module;
[0083] Step S6, the interactive feature vector I in S5 mid After being normalized and processed by a multi-layer perceptron, it is used as a residual and combined with I mid to be spliced together to form a new spliced feature vector I' fu ;
[0084] Step S7, send I' in S6 fu into the detection head of yolov7 for object detection.
[0085] Step S8, adopt the backpropagation algorithm and the stochastic gradient descent method to reduce the overall prediction error L all to train the object detection model, and finally obtain the final RGB-IR feature fusion object detection model after multiple iterations of training;
[0086] Step S9, use the trained RGB-IR feature fusion object detection model to perform object detection on the open-source dataset. First, the RGB image and the corresponding IR image are constructed into four-channel input data, and normalized to a unified size (such as 640×640×4 pixels), and then sent into the trained model to detect the test results.
[0087] As can be seen from the above embodiments, the visible light-thermal imaging object detection method based on dual-stream network fusion of the present invention can improve the robustness and accuracy of object detection. On the one hand, the IR image can provide the spatial position information of the target, making up for the deficiencies of the RGB image in low-light or occluded environments; on the other hand, the RGB image can provide rich texture and detail information. Through the feature extraction and fusion of the dual-stream network, the complementary characteristics of these two modalities are fully utilized, thereby effectively improving the performance of object detection. Specifically, first, the RGB image and the IR image are constructed into four-channel input data and input into the dual-stream fusion feature extraction neural network, which includes an RGB feature extraction stream and an IR feature extraction stream. The RGB feature extraction stream is mainly used to extract detailed features to make up for the deficiencies in details of the IR image; the IR feature extraction stream is used to obtain spatial information and enhance the description of the target position information. Next, the features of the two modalities are weighted and fused through the multi-head attention mechanism to generate a fused interactive feature vector. Finally, the fused features are optimized by using region-level triple constraints, so that the target features are close to each other in the feature space and far from the background features, thereby suppressing the interference of background noise and improving the detection accuracy.
[0088] That is, the present invention proposes a target detection method for visible light (RGB) images and infrared thermal imaging (IR) images based on the fusion of two-stream networks. This method uses a two-stream deep convolutional neural network framework. First, feature extraction networks for RGB and IR images are designed respectively. Then, the RGB image and the IR image are input into the corresponding feature extraction networks to construct the RGB stream and the IR stream respectively, extract the features of the two modalities, and synthesize a feature representation with richer features through a feature fusion module. In this method, the introduced IR image can provide the thermal radiation information of the target, while the RGB image provides rich texture and color information. The complementary characteristics of the two modalities provide diverse data sources for the network. Through the synergistic effect of the two-stream network structure and the feature fusion module, the network can focus on the key regions of the target during the learning process while suppressing background interference. This method has extremely strong robustness to diverse background changes and can effectively improve the accuracy and robustness of target detection in complex environments (such as low light, strong light, or cluttered backgrounds).
[0089] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by orientation words such as "front, back, up, down, left, right", "lateral, vertical, perpendicular, horizontal" and "top, bottom", etc. is usually based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description. Without contrary description, these orientation words do not indicate and imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation on the protection scope of the present invention; the orientation words "inside, outside" refer to the inside and outside relative to the contour of each component itself.
[0090] For the convenience of description, spatial relative terms such as "above...", "above...", "on the upper surface of...", "above" can be used here to describe the spatial positional relationship between a device or feature shown in the figure and other devices or features. It should be understood that the spatial relative terms are intended to include different orientations in use or operation in addition to the orientation described in the figure for the device. For example, if the device in the figure is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "under other devices or structures" afterwards. Thus, the exemplary term "above..." can include both the orientations of "above..." and "below...". The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and corresponding interpretations are made for the spatial relative descriptions used here.
[0091] In addition, it should be noted that the use of terms such as "first" and "second" to define components is only for the convenience of distinguishing the corresponding components. Without additional declaration, the above terms have no special meaning, and therefore should not be construed as a limitation on the protection scope of the present invention.
[0092] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.
Claims
1. A visible light-thermal imaging target detection method based on dual-stream network fusion, characterized in that, The method includes: Constructing the paired RGB images and corresponding IR images in the dataset into four-channel input data, where the RGB images have three channels and the IR images have a single channel; Normalizing the input data to a predetermined size and transmitting the normalized input data to a two-stream fusion feature extraction neural network to extract RGB features and IR features respectively; The RGB feature extractor of the dual-stream fusion feature extraction neural network extracts features from the visible light image to obtain the RGB feature map F rgb ; The IR feature extractor of the dual-stream fusion feature extraction neural network extracts the spatial information of the infrared image to obtain the IR feature map F ir ; Flatten the RGB feature map F rgb and the IR feature map F ir and then concatenate them in parallel after flattening to obtain the concatenated feature vector I fu ; Concatenated Feature Vector I fu It is successively processed by 3 multi-layer perceptrons to generate the corresponding query matrix Q, key matrix K, and value matrix V, and the generated query matrix Q, key matrix K, and value matrix V are input into the multi-head attention module to calculate the interaction feature vector I after feature fusion mid ; The interaction feature vector I after feature fusion mid After being normalized and processed by a multi-layer perceptron, it is used as a residual and the interaction feature vector I after feature fusion mid Spliced together to form a new spliced feature vector I' fu ; Perform object detection on the new spliced feature vector I’ through the detection head fu ; The backpropagation algorithm and the stochastic gradient descent method are used to reduce the overall prediction error L all to train the object detection model, and the final RGB-IR feature fusion object detection model is obtained after multiple iterations of training; Performing object detection on an open-source dataset using the final RGB-IR feature fusion object detection model.
2. The method according to claim 1, characterized in that splicing Eigenvector I fu is as follows: I fu = f Concat (f flatten (F rgb ), f flatten (F ir )) Among them, f Concat is the parallel splicing function, and f flatten (F rgb ) is the flattened RGB feature map, and f flatten (F ir ) is the flattened IR feature map.
3. The method according to claim 2, wherein The query matrix Q, key matrix K, and value matrix V are respectively: Q = MLP Q (I fu ), K = MLP K (I fu ), V = MLP V (I fu ), Among them, MLP is a multi-layer perceptron module, MLP Q and MLP K and MLP V respectively represent 3 non-reused multi-layer perceptrons.
4. The method according to claim 3, characterized in that, The interactive feature vector I after feature fusion mid is as follows: I mid = MHA(Q, K, V), where MHA is the multi-head attention module.
5. The method according to any one of claims 1-4, characterized in that, The predetermined size is 640×640×4 pixels.
6. The method according to any one of claims 1-5, characterized in that, The Resnet module of the RGB feature extractor includes a MaxPool layer, a 1x1Conv layer, and a 3x3Conv layer connected in sequence.
7. The method according to any one of claims 1-5, characterized in that, The Resnet module of the IR feature extractor includes a MaxPool layer and two consecutive 1x1Conv layers.
Citation Information
Cited By
Axle surface defect detection system for off-highway wide-body mining vehicle
CN122042683A