Infrared and visible light fusion recognition system and method based on modal difference feature guidance

Abstract

The present invention discloses an infrared-visible light fusion recognition system based on modal difference feature guidance, including a dual-stream backbone network module, an illumination heat perception module and a cascade region proposal module, wherein the dual-stream backbone network module includes an infrared feature extraction unit, a visible light feature extraction unit and a feature guidance unit, wherein the infrared feature extraction unit obtains infrared features of different scales of the original infrared image, and the visible light feature extraction unit obtains visible light features of different scales of the visible light image; the feature guidance unit obtains weighted infrared features and weighted visible light features; the illumination heat perception module obtains the credibility weights of the visible light features and the credibility weights of the infrared features; the cascade region proposal module is used to obtain the recognition result of the target. The present invention adopts an inter-level feature generation method of supplementary learning guided by inter-modal feature differences, which improves the dependency of inter-modal feature representation and improves the generalization ability of the network system.

Description

Technical Field

[0001] The present invention belongs to the technical field of multispectral image recognition, and in particular relates to an infrared and visible light fusion detection and recognition method guided by modal difference features. Background Art

[0002] Infrared images can capture clear target contour information, while visible light images can present detailed visual information of the target, including color, texture, structure, etc. In the case of insufficient light, it is essential to effectively utilize the complementary information of the two bands to achieve target detection.

[0003] In the early exploration of infrared-visible light fusion target detection, many scholars focused on studying imaging hardware and calibration to obtain paired infrared-visible light image sequences, and used machine learning detection methods to design dual-band fusion detectors to verify the effect of the data set. Hwang S, Park J, Kim N and others built an infrared-visible light paired imaging system consisting of a color camera, a thermal imager, a beam splitter and a three-axis camera fixture, and designed a dual-band target detector based on ACF (visible light channel feature) + T (pixel intensity) + THOG (histogram of gradient orientation) features and Adaboost classifiers. The classification results were obtained by extracting ACF and infrared pixel intensity and THOG features and fusing them.

[0004] With the rapid development of deep neural networks in the field of visible light image detection, researchers are committed to applying deep neural networks to dual-band feature fusion detection. In view of the field of view and modality differences between infrared and visible light modalities, Zhang L, Zhu X, Chen X and others designed a new alignment region CNN network to process weakly aligned data end-to-end. For a set of input color images and heat maps, after feature extraction, the input region features are aligned using quantity prediction and region of interest matching, and then the aligned image features are merged and confidence fused. The above methods alleviate the differences between modalities, but do not further start from different feature layers, complement each other through the differences between modal features, and guide the generation of current features. At the same time, in the process of feature fusion, the current method has not established a complete fusion guidance strategy, resulting in low fusion detection accuracy. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides an infrared and visible light fusion detection and recognition method based on modal difference feature guidance. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The present invention provides an infrared-visible light fusion recognition system based on modal difference feature guidance, including a dual-stream backbone network module, a light heat perception module and a cascade region proposal module, wherein:

[0007] The dual-stream backbone network module includes an infrared feature extraction unit, a visible light feature extraction unit and a feature guidance unit, wherein the infrared feature extraction unit is used to obtain infrared features of different scales of the original infrared image, and the visible light feature extraction unit is used to obtain visible light features of different scales of the visible light image of the same scene as the infrared image; the feature guidance unit is used to fuse and weight the infrared features and visible light features of the current scale to obtain weighted infrared features and weighted visible light features, so as to cross-guide the formation of the infrared features and visible light features of the next scale in the infrared feature extraction unit and the visible light feature extraction unit;

[0008] The light heat perception module is used to obtain the credibility weight of the visible light feature and the credibility weight of the infrared feature according to the image heat value of the original infrared image and the image illumination value of the original visible light image;

[0009] The cascade region proposal module is used to obtain a target recognition result according to infrared features of different scales, visible light features of different scales, and the credibility weights of the visible light features and the credibility weights of the infrared features.

[0010] In one embodiment of the present invention, the infrared feature extraction unit includes a plurality of cascaded infrared feature extraction convolutional layers to obtain infrared light features of different scales, the visible light feature extraction unit includes a plurality of cascaded visible light feature extraction units to obtain visible light features of different scales, and the feature guidance unit includes a plurality of cascaded modal difference guidance subunits, wherein:

[0011] The modal difference guiding subunit uses the infrared light features and visible light features of the current scale and the infrared-visible light fusion vector obtained by the previous modal difference guiding subunit to obtain weighted infrared features, weighted visible light features and infrared-visible light fusion vectors, and transmits the weighted infrared features to the visible light feature extraction unit of the next scale, and transmits the weighted visible light features to the infrared feature extraction unit of the next scale to guide the formation of the infrared features and visible light features of the next scale in the infrared feature extraction unit and the visible light feature extraction unit; and transmits the infrared-visible light fusion vector to the next modal difference guiding subunit.

[0012] In one embodiment of the present invention, the modality difference guiding subunit is specifically used for:

[0013] The current visible light feature, the current infrared feature and the visible light infrared fusion feature output by the previous modal difference guided subunit are fused, and feature separation is performed through three parallel 3×3 convolutional layers to obtain visible light features, infrared features and visible light infrared fusion features that include more semantic information; then the Sigmoid activation function is used to obtain the weights corresponding to the current visible light feature and the current infrared feature, thereby forming weighted visible light features and weighted infrared features.

[0014] In one embodiment of the present invention, the infrared feature extraction unit specifically includes a first base layer, a first dense fusion layer, a second dense fusion layer, a third dense fusion layer and a fourth dense fusion layer, the visible light feature extraction unit specifically includes a second base layer, a fifth dense fusion layer, a sixth dense fusion layer, a seventh dense fusion layer and an eighth dense fusion layer, and the feature guidance unit includes a first modal feature difference guidance subunit, a second modal feature difference guidance subunit, a third modal feature difference guidance subunit and a fourth modal feature difference guidance subunit, wherein,

[0015] The first base layer is used to input an original visible light image and obtain a first-scale visible light feature, the second base layer is used to input an original infrared image and obtain a first-scale infrared feature, and the first modal feature difference guiding subunit is used to fuse the first-scale visible light feature and the first-scale infrared feature, and obtain a first-scale infrared-visible light fusion vector, a first weighted infrared feature, and a first weighted visible light feature;

[0016] The first dense fusion layer is used to obtain a second-scale visible light feature by using the first weighted infrared feature and the first-scale visible light feature; the fifth dense fusion layer is used to obtain a second-scale infrared feature by using the first weighted visible light feature and the first-scale infrared feature; the second modal feature difference guiding subunit is used to fuse the second-scale visible light feature, the second-scale infrared feature and the first-scale infrared-visible light fusion vector to obtain a second-scale infrared-visible light fusion vector, a second weighted infrared feature and a second weighted visible light feature;

[0017] The second dense fusion layer is used to obtain a third-scale visible light feature by using the second weighted infrared feature and the second-scale visible light feature; the sixth dense fusion layer is used to obtain a third-scale infrared feature by using the second weighted visible light feature and the second-scale infrared feature; the third modal feature difference guiding subunit is used to fuse the third-scale visible light feature, the third-scale infrared feature and the second-scale infrared-visible light fusion vector to obtain a third-scale infrared-visible light fusion vector, a third weighted infrared feature and a third weighted visible light feature;

[0018] The third dense fusion layer is used to obtain a fourth-scale visible light feature by using the third weighted infrared feature and the third-scale visible light feature; the seventh dense fusion layer is used to obtain a fourth-scale infrared feature by using the third weighted visible light feature and the third-scale infrared feature; the fourth modal feature difference guiding subunit is used to fuse the fourth-scale visible light feature, the fourth-scale infrared feature and the third-scale infrared-visible light fusion vector to obtain a fourth-scale infrared-visible light fusion vector, a fourth weighted infrared feature and a fourth weighted visible light feature;

[0019] The fourth dense fusion layer is used to obtain a fifth-scale visible light feature using the fourth weighted infrared feature and the fourth-scale visible light feature; the eighth dense fusion layer is used to obtain a fifth-scale infrared feature using the fourth weighted visible light feature and the fourth-scale infrared feature.

[0020] In one embodiment of the present invention, the first base layer and the second base layer are both composed of a 5×5 convolution kernel with a step size of 1, a 1×1 convolution kernel with a step size of 1, and a 3×3 convolution kernel with a step size of 2; the first dense fusion layer, the second dense fusion layer, the third dense fusion layer, the fourth dense fusion layer, the fifth dense fusion layer, the sixth dense fusion layer, the seventh dense fusion layer and the eighth dense fusion layer are all composed of a shullflenetv2 basic convolution kernel with a step size of 2.

[0021] In one embodiment of the present invention, the light and heat perception module includes a light and heat perception unit, a light perception mechanism unit and a heat perception mechanism unit, wherein:

[0022] The light and heat sensing unit is used to process the input original visible light image and original infrared image to obtain the minimum light value and the maximum light value of the target area, as well as the minimum heat value and the maximum heat value of the target area;

[0023] The illumination perception mechanism unit is used to obtain the visible light feature credibility weight and the infrared feature credibility weight under the current illumination conditions according to the minimum illumination value and the maximum illumination value of the target area;

[0024] The heat perception mechanism unit is used to obtain the visible light feature credibility weight and the infrared feature credibility weight under the current heat condition according to the minimum heat value and the maximum heat value of the target area.

[0025] In one embodiment of the present invention, the light heat perception unit includes two convolutional layers and three fully connected layers. After the convolutional layers, a RELU activation function and a 2×2 maximum pooling layer are used to compress and extract features. The light intensity loss of the light heat perception unit is:

[0026]

[0027] Among them, w d and w n are the minimum and maximum illumination values ​​of the target area acquired by the illumination and heat sensing unit, and The actual labels for day and night,

[0028] The infrared intensity loss of the light and heat sensing unit for:

[0029]

[0030] Among them, m d and m n are the lowest and highest heat values ​​of the target area acquired by the light heat sensing unit, and The true labels for cold and hot.

[0031] In one embodiment of the present invention, the cascade region proposal module includes a feature fusion unit, a classification regression unit and a parameter adjustment unit, wherein:

[0032] The feature fusion unit is used to fuse the visible light features of the third scale, the visible light features of the fourth scale, the visible light features of the fifth scale, the infrared features of the third scale, the infrared features of the fourth scale, and the infrared features of the fifth scale to obtain a fused image;

[0033] The classification regression unit is used to perform probability prediction on each target type in the fused image to obtain a target category approximation estimate, and calculate the intersection-over-union ratio of each target position prediction frame and the true value frame to obtain a target position offset estimate;

[0034] The parameter adjustment unit adjusts the target category approximation estimation value and the target position offset estimation value through the thermal illumination perception parameters output by the illumination thermal perception module, and obtains the category prediction structure and position prediction result of the target in the image.

[0035] Another aspect of the present invention provides an infrared-visible light fusion recognition method guided by modal difference features, comprising:

[0036] S1: constructing an infrared and visible light fusion recognition system guided by modal difference features as described in any one of the above embodiments;

[0037] S2: training the dual-stream backbone network module, the light heat perception module and the cascade region proposal module in the infrared-visible light fusion recognition system to obtain a trained infrared-visible light fusion recognition system;

[0038] S3: Input the original infrared image and the original visible light image of the same scene into the trained infrared-visible light fusion recognition system to obtain the target type and target position in the image.

[0039] In one embodiment of the present invention, the S2 includes:

[0040] A first training data set is formed by using a large number of pictures with thermal value labels and lighting value labels to train the light thermal perception module to obtain a trained light thermal perception module;

[0041] The infrared-visible light fusion recognition system is constructed using the original dual-stream backbone network module, the trained light heat perception module and the original cascade region proposal module, and the original dual-stream backbone network module and the original cascade region proposal module in the infrared-visible light fusion recognition system are trained using a second training data set formed by a large number of images with target position labels and target type labels to obtain a trained infrared-visible light fusion recognition system.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. The present invention adopts an inter-level feature generation method that uses inter-modal feature differences to guide supplementary learning. By densely embedding modal feature difference guidance units between feature layers, the reliability weight of each modal feature is obtained, and then deep integration of visible light and infrared modal features is performed, avoiding the problem of false detection caused by simple fusion strategies of linear combination or splicing that retain erroneous information during the fusion process.

[0044] 2. The present invention integrates different feature layers and different scale features to improve the detection performance of multi-scale targets. The low-level features have higher resolution and contain more position and detail information. They are less semantic and have more noise. The high-level features have stronger semantic information, but have low resolution and poor perception of details. The use of multi-level features is beneficial to the detection capability of targets of different sizes.

[0045] 3. The present invention adopts infrared thermal intensity and visible light illumination perception modules to obtain the illumination value and thermal intensity value in the current scene, and adaptively adjusts the confidence scores and position offset scores of the infrared modal branch and the visible light modal branch, thereby improving the network's generalization ability for different scenes, weather and targets.

[0046] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1It is a structural block diagram of an infrared and visible light fusion recognition system based on modal difference feature guidance provided by an embodiment of the present invention;

[0048] Figure 2 It is a specific structural diagram of an infrared-visible light fusion recognition system based on modal difference feature guidance provided by an embodiment of the present invention;

[0049] Figure 3 is a structural schematic diagram of a modal difference guiding subunit provided by an embodiment of the present invention;

[0050] Figure 4 It is a schematic diagram of the working process of an infrared-visible light fusion recognition system guided by modal difference features provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following is a detailed description of an infrared-visible light fusion recognition system and method based on modal difference feature guidance proposed by the present invention, in combination with the accompanying drawings and specific implementation methods.

[0052] The above and other technical contents, features and effects of the present invention are clearly presented in the following detailed description of the specific implementation modes in conjunction with the accompanying drawings. Through the description of the specific implementation modes, the technical means and effects adopted by the present invention to achieve the predetermined purpose can be more deeply and specifically understood. However, the attached drawings are only for reference and explanation purposes and are not used to limit the technical solutions of the present invention.

[0053] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants are intended to cover non-exclusive inclusion, so that an article or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed. In the absence of more restrictions, the elements defined by the statement "including one..." do not exclude the existence of other identical elements in the article or device including the elements.

[0054] Embodiment 1

[0055] See also Figure 1 and Figure 2 , Figure 1 It is a structural block diagram of an infrared and visible light fusion recognition system based on modal difference feature guidance provided by an embodiment of the present invention; Figure 2It is a specific structural diagram of an infrared-visible light fusion recognition system based on modal difference feature guidance provided by an embodiment of the present invention. The infrared-visible light fusion recognition system includes a dual-stream backbone network module 1, a light heat perception module 2, and a cascade region proposal module 3. The dual-stream backbone network module 1 includes an infrared feature extraction unit 11, a visible light feature extraction unit 12, and a feature guidance unit 13, wherein the infrared feature extraction unit 11 is used to obtain infrared features of different scales of the original infrared image, and the visible light feature extraction unit 12 is used to obtain visible light features of different scales of the visible light image of the same scene as the original infrared image; the feature guidance unit 13 is used to fuse and weight the infrared features and visible light features of the current scale to obtain weighted infrared features and weighted visible light features, so as to cross-guide the formation of the next scale infrared features and visible light features in the infrared feature extraction unit 11 and the visible light feature extraction unit 12.

[0056] The light heat perception module 2 is used to obtain the credibility weight of the visible light feature and the credibility weight of the infrared feature according to the image heat value of the original infrared image and the image illumination value of the original visible light image; the cascade region proposal module 3 is used to obtain the recognition result of the target according to the infrared features of different scales, the visible light features of different scales, and the credibility weight of the visible light feature and the credibility weight of the infrared feature.

[0057] Further, the infrared feature extraction unit 11 in the dual-stream backbone network module 1 includes a plurality of cascaded infrared feature extraction convolutional layers to obtain infrared light features of different scales, and the visible light feature extraction unit 12 includes a plurality of cascaded visible light feature extraction units to obtain visible light features of different scales. The feature guidance unit 13 includes a plurality of cascaded modal difference guidance subunits, wherein the modal difference guidance subunit uses the infrared light features and visible light features of the current scale and the infrared-visible light fusion vector obtained by the previous modal difference guidance subunit to obtain weighted infrared features, weighted visible light features, and infrared-visible light fusion vectors, and transmits the weighted infrared features to the visible light feature extraction unit of the next scale, and transmits the weighted visible light features to the infrared feature extraction unit of the next scale to guide the formation of the infrared features and visible light features of the next scale in the infrared feature extraction unit and the visible light feature extraction unit; and transmits the infrared-visible light fusion vector to the next modal difference guidance subunit.

[0058] For further information, see Figure 3 , the modal difference guiding subunit is specifically used for:

[0059] The current visible light feature, the current infrared feature and the visible light infrared fusion feature output by the previous modal difference guided subunit are fused, and feature separation is performed through three parallel 3×3 convolutional layers to obtain visible light features, infrared features and visible light infrared fusion features that include more semantic information; then the Sigmoid activation function is used to obtain the weights corresponding to the current visible light feature and the current infrared feature, thereby forming a weighted visible light feature and a weighted infrared feature.

[0060] For further information, see Figure 2 The infrared feature extraction unit 11 of this embodiment specifically includes a first base layer, a first dense fusion layer, a second dense fusion layer, a third dense fusion layer and a fourth dense fusion layer. The visible light feature extraction unit 12 specifically includes a second base layer, a fifth dense fusion layer, a sixth dense fusion layer, a seventh dense fusion layer and an eighth dense fusion layer. The feature guidance unit includes 13 a first modal feature difference guidance subunit, a second modal feature difference guidance subunit, a third modal feature difference guidance subunit and a fourth modal feature difference guidance subunit, wherein,

[0061] The first base layer is used to input an original visible light image and obtain a first-scale visible light feature, the second base layer is used to input an original infrared image and obtain a first-scale infrared feature, and the first modal feature difference guiding subunit is used to fuse the first-scale visible light feature and the first-scale infrared feature, and obtain a first-scale infrared-visible light fusion vector, a first weighted infrared feature, and a first weighted visible light feature;

[0062] The first dense fusion layer is used to obtain a second-scale visible light feature by using the first weighted infrared feature and the first-scale visible light feature; the fifth dense fusion layer is used to obtain a second-scale infrared feature by using the first weighted visible light feature and the first-scale infrared feature; the second modal feature difference guiding subunit is used to fuse the second-scale visible light feature, the second-scale infrared feature and the first-scale infrared-visible light fusion vector to obtain a second-scale infrared-visible light fusion vector, a second weighted infrared feature and a second weighted visible light feature;

[0063] The second dense fusion layer is used to obtain a third-scale visible light feature by using the second weighted infrared feature and the second-scale visible light feature; the sixth dense fusion layer is used to obtain a third-scale infrared feature by using the second weighted visible light feature and the second-scale infrared feature; the third modal feature difference guiding subunit is used to fuse the third-scale visible light feature, the third-scale infrared feature and the second-scale infrared-visible light fusion vector to obtain a third-scale infrared-visible light fusion vector, a third weighted infrared feature and a third weighted visible light feature;

[0064] The third dense fusion layer is used to obtain a fourth-scale visible light feature by using the third weighted infrared feature and the third-scale visible light feature; the seventh dense fusion layer is used to obtain a fourth-scale infrared feature by using the third weighted visible light feature and the third-scale infrared feature; the fourth modal feature difference guiding subunit is used to fuse the fourth-scale visible light feature, the fourth-scale infrared feature and the third-scale infrared-visible light fusion vector to obtain a fourth-scale infrared-visible light fusion vector, a fourth weighted infrared feature and a fourth weighted visible light feature;

[0065] The fourth dense fusion layer is used to obtain a fifth-scale visible light feature using the fourth weighted infrared feature and the fourth-scale visible light feature; the eighth dense fusion layer is used to obtain a fifth-scale infrared feature using the fourth weighted visible light feature and the fourth-scale infrared feature.

[0066] Specifically, the infrared feature extraction unit and the visible light feature extraction unit in the dual-stream backbone network module of this embodiment respectively include five convolutional layers (i.e., one base layer and four dense fusion layers) and embed a modal difference feature guidance subunit layer by layer. First, the original infrared image and the visible light image are respectively feature extracted through the convolutional layer to obtain visible light features and infrared features. Then, the visible light features, infrared features, and the infrared-visible light fusion features after the infrared light features and visible light features of the previous layer are fused into the current modal difference feature guidance subunit. Through cross-guidance, the weighted infrared features are output to guide the generation of the next level of visible light features, and the weighted visible light features are output to guide the generation of the next level of infrared features.

[0067] Furthermore, the five convolutional layers in the infrared feature extraction unit and the visible light feature extraction unit include a base layer and four dense fusion layers, the base layer is composed of a 5×5 depth-wise convolution kernel with a step size of 1, a 1×1 convolution kernel with a step size of 1, and a 3×3 convolution kernel with a step size of 2, and the dense fusion layer is composed of a shullflenetv2 basic convolution kernel with a step size of 2. In this embodiment, the first base layer and the second base layer are both composed of a 5×5 convolution kernel with a step size of 1, a 1×1 convolution kernel with a step size of 1, and a 3×3 convolution kernel with a step size of 2; the first dense fusion layer, the second dense fusion layer, the third dense fusion layer, the fourth dense fusion layer, the fifth dense fusion layer, the sixth dense fusion layer, the seventh dense fusion layer, and the eighth dense fusion layer are all composed of a shullflenetv2 basic convolution kernel with a step size of 2. After the original visible light image of size H×W×C (H is the height of the image, W is the width of the image, and C is the number of channels) is input into the visible light feature extraction unit, during the hierarchical convolution process, visible light feature maps of different scales of H / 2×W / 2×32, H / 4×W / 4×64, H / 8×W / 8×64, H / 16×W / 16×128 and H / 32×W / 32×256 are gradually obtained. Similarly, after the original infrared image of size H×W×C (H is the height of the image, W is the width of the image, and C is the number of channels) is input into the infrared feature extraction unit, during the hierarchical convolution process, infrared feature maps of different scales of H / 2×W / 2×32, H / 4×W / 4×64, H / 8×W / 8×64, H / 16×W / 16×128 and H / 32×W / 32×256 are gradually obtained.

[0068] Further, the input of each modality difference guiding subunit in the feature guiding unit 13 includes the visible light feature extracted by the current visible light convolution layer, the infrared feature extracted by the current infrared convolution layer, and the visible light infrared fusion feature obtained in the previous modality difference guiding subunit. The modality difference guiding subunit fuses the current visible light feature, the current infrared feature, and the visible light infrared fusion feature output by the previous modality difference guiding subunit, and separates the three features through three parallel 3×3 convolutions to increase the receptive field, so as to obtain higher-level visible light features, infrared features, and visible light infrared fusion features including more semantic information (i.e., obtaining the association information between the target and the surrounding background), and according to the contribution ratio of each modality (visible light modality or infrared modality) in the dual-stream backbone network, the Sigmoid activation function is used to obtain the weight corresponding to each modality feature, and the weighted infrared feature and weighted visible light feature are obtained. Then, the output weighted infrared feature is returned to the visible light feature stream for splicing. The output weighted visible light feature is returned to the infrared feature stream for splicing. At the same time, the output infrared-visible light fusion features are input into the next-level feature difference guidance subunit to participate in the next-level feature generation.

[0069] Specifically, before adding the feature guidance unit, the layer-by-layer feature generation process in the dual-stream backbone network module can be expressed as:

[0070]

[0071] in, represents the visible light feature of the lth layer, represents the l+1th layer visible light feature. f(x, l) represents the convolution merge and activation operation of the lth layer. According to the chain rule, the visible light back propagation formula of the corresponding loss function can be expressed as:

[0072]

[0073] Correspondingly, the back propagation formula of infrared features can be expressed as:

[0074]

[0075] in, represents the infrared characteristics of layer l, represents the infrared features of the l+1 layer. f(x,l) represents the convolutional merging and activation operation of the lth layer.

[0076] Furthermore, after densely adding feature difference guidance subunits in the dual-stream backbone network module, the weighted visible light features are used to guide the generation of the next level of infrared features to complete the cross-guidance between modalities. The inter-level visible light feature generation mode can be expressed as:

[0077]

[0078] in, represents the l-layer visible light feature, represents the l+1 layer of visible light characteristics, represents the weighted feature obtained after the infrared feature and the visible light feature pass through the weight perception module, τ(·) represents the guidance of feature addition or splicing, and the corresponding back propagation according to the chain rule can be expressed as:

[0079]

[0080]

[0081] in, represents the weighted features obtained after the infrared features and visible light features pass through the weight perception module, and ψ(·) represents the weighted operation of the visible light infrared features. In the feature difference guidance subunit, the generation of visible light features includes the joint participation of infrared weighted features and visible light features, and the same is true for infrared weighted features. The cross-difference guidance mechanism generated by the feature guidance unit can deepen the connection between multimodal networks and establish deeper and more accurate modal correlations, thereby obtaining more discriminative enhanced features.

[0082] Furthermore, the light and heat perception module 2 of the present embodiment includes a light and heat perception unit 21, a light perception mechanism unit 22 and a heat perception mechanism unit 23, wherein the light and heat perception unit 21 is used to process the input original visible light image and the original infrared image to obtain the minimum light value and the maximum light value of the target area, as well as the minimum heat value and the maximum heat value of the target area; the light perception mechanism unit 22 is used to obtain the visible light feature credibility weight and the infrared feature credibility weight under the current light conditions according to the minimum light value and the maximum light value of the target area; the heat perception mechanism unit 23 is used to obtain the visible light feature credibility weight and the infrared feature credibility weight under the current heat conditions according to the minimum heat value and the maximum heat value of the target area.

[0083] Specifically, see Figure 2 , this embodiment uses a tiny neural network as a light and heat perception unit to capture the illumination value of the visible light image and the heat intensity of the infrared image. In order to reduce the computational complexity, the size of the visible light and infrared images is resized to 56×56 and fed into the light and heat perception module. The light and heat perception unit of this embodiment includes two convolutional layers and three fully connected layers. After the convolutional layers, a RELU activation function and a 2×2 maximum pooling layer are used to compress and extract features, and the network is optimized by optimizing the illumination loss and the cross entropy loss function of the heat. The light intensity loss of the light and heat perception unit is:

[0084]

[0085] Among them, w d and w n is the Softmax output of the fully connected layer, which are the lowest and highest illumination values ​​of the target area obtained by the illumination heat sensing unit, respectively. and Real labels indicating day and night.

[0086] The illumination perception mechanism unit 22 is used to obtain the visible light feature credibility weight and the infrared feature credibility weight under the current illumination conditions according to the minimum illumination value and the maximum illumination value of the target area. The calculation formula is:

[0087]

[0088] w t =1-w r

[0089] Among them, w r represents the credibility weight of the visible light feature under the current lighting conditions, w t represents the infrared feature credibility weight under the current lighting conditions. In order to adapt in the network, w is readjusted in the lighting perception mechanism unit 22. d and w n , where w is a parameter set according to the actual situation, |w|∈[0,1] is an independent prediction of the deviation starting from 0.5, and a w , r w are two learnable parameters initialized with 1 and 0.

[0090] Furthermore, the infrared intensity loss Lt of the light heat sensing unit i for:

[0091]

[0092] Among them, m d and m n is the Softmax output of the fully connected layer, which are the lowest and highest heat values ​​of the target area obtained by the light heat perception unit, respectively. and The true labels for cold and hot.

[0093] The heat perception mechanism unit 23 is used to obtain the visible light feature credibility weight and the infrared feature credibility weight under the current heat condition according to the minimum heat value and the maximum heat value of the target area. The calculation formula is:

[0094]

[0095] Among them, m r Represents the credibility weight of the visible light feature under the current thermal condition, m t represents the infrared feature credibility weight under the current heat condition. Specifically, in order to adapt in the network, the heat perception mechanism unit 23 readjusts m h and m c , where m is a parameter set according to the actual situation, |m|∈[0,1] is an independent prediction of the deviation starting from 0.5, and a m , r m are two learnable parameters initialized with 1 and 0.

[0096] Furthermore, the cascade region proposal module 3 of the present embodiment includes a feature fusion unit, a classification regression unit and a parameter adjustment unit, wherein the feature fusion unit is used to fuse the visible light features of the third scale, the visible light features of the fourth scale, the visible light features of the fifth scale, the infrared features of the third scale, the infrared features of the fourth scale and the infrared features of the fifth scale to obtain a fused image. The classification and regression unit first performs probability prediction on each possible type of the fused image through the softmax function of the fully connected layer to obtain a target category approximation estimate; then calculates the intersection over union (IOU) of the target position prediction box and the true value box to obtain a target position offset estimate; the parameter adjustment unit adjusts the target category approximation estimate and the target position offset estimate through the thermal illumination perception parameters obtained by the illumination thermal perception module (i.e., the visible light feature credibility weight and infrared feature credibility weight under the current illumination conditions, and the visible light feature credibility weight and infrared feature credibility weight under the current thermal conditions); when the final target category approximation estimate and IOU are both higher than 75%, it is considered that the current category prediction and position prediction are accurate, and the image detection and recognition result is obtained.

[0097] Specifically, due to the large range of target size variation, the multi-level features extracted by the dual-stream backbone network module are first fused, and the target parameters are obtained by cascading region proposals. Specifically, the visible light features of the third scale, the visible light features of the fourth scale, the visible light features of the fifth scale, the infrared features of the third scale, the infrared features of the fourth scale, and the infrared features of the fifth scale are fused to obtain a fused image, and then the fused image is subjected to a probability prediction of each possible type through the softmax function of the fully connected layer to obtain the target category approximation estimate s0; then the intersection-over-union (IOU) of the target position prediction box and the true value box is calculated to obtain the target position offset estimate t0.

[0098] Subsequently, the heat illumination perception parameters obtained by the illumination heat perception module, that is, the visible light feature credibility weight and the infrared feature credibility weight under the current illumination conditions, and the visible light feature credibility weight and the infrared feature credibility weight under the current heat conditions are used to adjust the target category approximation estimate and the target position offset estimate to obtain the adjusted values ​​s1 and t1, that is, the illumination heat parameter w r 、w t 、m r 、m t Regression offset t for visible light and infrared feature predictions r ,t t and the confidence score s r 、s t Re-weight and obtain the adjusted values ​​s1 and t1. The specific calculation formula is:

[0099]

[0100]

[0101] Among them, s r and t r is the confidence score and regression offset of the visible light feature prediction of different scales obtained by the visible light feature extraction unit; t and t t Confidence scores and regression offsets of infrared feature predictions of different scales obtained using the infrared feature extraction unit.

[0102] Final confidence score s fina and the regression offset t fina The calculation formula is as follows:

[0103]

[0104]

[0105] The final confidence score is obtained by multiplying the two-stage scores s0 and s1. When both stage scores are high, they are used to motivate the final confidence score. The target position parameters are approximated to the target boundary using the summation method. cls A focal weight is added to solve the positive and negative imbalance problem. The classification loss L in this embodiment cls It is expressed as:

[0106]

[0107] Here, α=0.3 and γ=2 are set. i is the confidence score of target i, that is, the final confidence score s above fina The total loss is the light loss Heat loss Classification loss L cls and the regression loss L reg The sum of the regression loss L reg is the smooth L1 loss proposed by Faster-RCNN. Specifically, the regression loss is expressed as:

[0108]

[0109] in, represents the ground truth bounding box vector of sample i, t i Represents the approximate position estimate of target i, that is, the above regression offset t fina , R represents the smooth L1 function in Faster-Rcnn.

[0110] The total loss is the classification loss L cls and the regression loss L reg The total loss function L is as follows:

[0111] L=L cls +L reg .

[0112] It should be noted that, in the actual process, after constructing the infrared-visible light fusion recognition system guided by modal difference features, it is necessary to first train the system to obtain the trained system before target recognition can be performed on the image to be recognized.

[0113] During the training process, first, a large number of pictures with heat value labels and lighting value labels are used to form a first training data set, and the light heat perception module is trained until the light intensity loss of the light heat perception unit and the infrared intensity loss value of the light heat perception unit meet the predetermined requirements, and the trained light heat perception module is obtained; then, the infrared-visible light fusion recognition system is constructed using the original dual-stream backbone network module, the trained light heat perception module and the original cascade region proposal module, and the second training data set formed by a large number of pictures with target position labels and target type labels is used to train the original dual-stream backbone network module and the original cascade region proposal module in the infrared-visible light fusion recognition system until the above-mentioned total loss function L meets the predetermined requirements, and a trained infrared-visible light fusion recognition system is obtained.

[0114] Subsequently, the original infrared image and the original visible light image of the same scene are input into the trained infrared-visible light fusion recognition system to obtain the target type and target position information in the image.

[0115] The embodiment of the present invention adopts an inter-level feature generation method that guides supplementary learning based on inter-modal feature differences, thereby improving the dependency of inter-modal feature representation. Different feature layers and different scale features are fused to improve the detection performance of multi-scale targets. Infrared thermal intensity and visible light illumination perception modules are used to weight the fused features, making the network more suitable for different scenes, weather and targets, and improving the generalization ability of the network. This embodiment fuses different feature layers and different scale features to improve the detection performance of multi-scale targets. Low-level features have higher resolution and contain more location and detail information. They have lower semantics and more noise. High-level features have stronger semantic information, but low resolution and poor perception of details. The use of multi-level features is beneficial to the detection ability of targets of different sizes.

[0116] Embodiment 2

[0117] Based on the above embodiment, this embodiment provides an infrared-visible light fusion recognition method guided by modal difference features, the method comprising:

[0118] S1: Construct an infrared-visible light fusion recognition system guided by modal difference features as described in Example 1.

[0119] The infrared visible light fusion recognition system of this embodiment includes a dual-stream backbone network module 1, a light heat perception module 2 and a cascade region proposal module 3. The dual-stream backbone network module 1 includes an infrared feature extraction unit 11, a visible light feature extraction unit 12 and a feature guidance unit 13, wherein the infrared feature extraction unit 11 is used to obtain infrared features of different scales of the original infrared image, and the visible light feature extraction unit 12 is used to obtain visible light features of different scales of the visible light image of the same scene as the original infrared image; the feature guidance unit 13 is used to cross-guide the infrared features and visible light features of the current scale to obtain weighted infrared features and weighted visible light features to guide the formation of the next scale infrared features and visible light features in the infrared feature extraction unit 11 and the visible light feature extraction unit 12.

[0120] The light heat perception module 2 is used to obtain the infrared image heat value in the original infrared image and the visible light image illumination value in the original visible light image; the cascade region proposal module 3 is used to obtain the target recognition result based on infrared features of different scales, visible light features of different scales, and infrared image heat value and visible light image illumination value.

[0121] S2: Train the dual-stream backbone network module, the light heat perception module and the cascade region proposal module in the infrared-visible light fusion recognition system to obtain a trained infrared-visible light fusion recognition system.

[0122] Further, the S2 includes:

[0123] A first training data set is formed by using a large number of pictures with thermal value labels and lighting value labels to train the light thermal perception module to obtain a trained light thermal perception module;

[0124] The infrared-visible light fusion recognition system is constructed using the original dual-stream backbone network module, the trained light heat perception module and the original cascade region proposal module, and the original dual-stream backbone network module and the original cascade region proposal module in the infrared-visible light fusion recognition system are trained using a second training data set formed by a large number of images with target position labels and target type labels to obtain a trained infrared-visible light fusion recognition system.

[0125] S3: Input the original infrared image and the original visible light image of the same scene into the trained infrared-visible light fusion recognition system to obtain the target type and target position in the image.

[0126] It should be noted that the working principle and data processing process of the infrared-visible light fusion recognition system can be found in Example 1 and will not be described in detail here.

[0127] Another embodiment of the present invention provides a storage medium, in which a computer program is stored, and the computer program is used to execute the steps of infrared-visible light fusion recognition based on modal difference feature guidance described in the above embodiment. Another aspect of the present invention provides an electronic device, including a memory and a processor, in which a computer program is stored, and when the processor calls the computer program in the memory, the steps of the infrared-visible light fusion recognition method based on modal difference feature guidance as described in the above embodiment are implemented. Specifically, the above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including several instructions for an electronic device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0128] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. An infrared and visible light fusion recognition system based on modal difference feature guidance, characterized in that: It includes a dual-stream backbone network module, a light heat perception module, and a cascade region proposal module, among which: The dual-stream backbone network module includes an infrared feature extraction unit, a visible light feature extraction unit and a feature guidance unit, wherein the infrared feature extraction unit is used to obtain infrared features of different scales of the original infrared image, and the visible light feature extraction unit is used to obtain visible light features of different scales of the visible light image of the same scene as the infrared image; the feature guidance unit is used to fuse and weight the infrared features and visible light features of the current scale to obtain weighted infrared features and weighted visible light features, so as to cross-guide the formation of the infrared features and visible light features of the next scale in the infrared feature extraction unit and the visible light feature extraction unit; The light heat perception module is used to obtain the credibility weight of the visible light feature and the credibility weight of the infrared feature according to the image heat value of the original infrared image and the image illumination value of the original visible light image; The cascade region proposal module is used to obtain the recognition result of the target according to the infrared features of different scales, the visible light features of different scales, the credibility weights of the visible light features and the credibility weights of the infrared features; The infrared feature extraction unit includes a plurality of cascaded infrared feature extraction convolutional layers to obtain infrared light features of different scales; the visible light feature extraction unit includes a plurality of cascaded visible light feature extraction units to obtain visible light features of different scales; The feature guiding unit includes a plurality of cascaded modal difference guiding subunits, wherein the modal difference guiding subunit obtains weighted infrared features, weighted visible light features and infrared visible light fusion vectors using infrared light features and visible light features of the current scale and the infrared visible light fusion vector obtained by the previous modal difference guiding subunit, and transmits the weighted infrared features to the visible light feature extraction unit of the next scale, and transmits the weighted visible light features to the infrared feature extraction unit of the next scale to cross-guide the formation of infrared features and visible light features of the next scale in the infrared feature extraction unit and the visible light feature extraction unit, and transmits the infrared visible light fusion vector to the next modal difference guiding subunit, The modal difference guiding subunit is specifically used for: The current visible light feature, the current infrared feature, and the visible light infrared fusion feature output by the previous modality difference guidance subunit are fused, and feature separation is performed through three parallel 3×3 convolutional layers to obtain visible light features, infrared features, and visible light infrared fusion features containing more semantic information. Subsequently, the Sigmoid activation function is used to obtain the weights corresponding to the current visible light feature and the current infrared feature, thereby forming weighted visible light features and weighted infrared features. The light and heat perception module includes a light and heat perception unit, a light perception mechanism unit and a heat perception mechanism unit, wherein: The light and heat sensing unit is used to obtain the minimum light value and the maximum light value of the target area, as well as the minimum heat value and the maximum heat value of the target area according to the input original visible light image and the original infrared image; The illumination perception mechanism unit is used to obtain the visible light feature credibility weight and the infrared feature credibility weight under the current illumination conditions according to the minimum illumination value and the maximum illumination value of the target area; The heat perception mechanism unit is used to obtain the visible light feature credibility weight and the infrared feature credibility weight under the current heat condition according to the minimum heat value and the maximum heat value of the target area; The thermal illumination perception parameters obtained by the illumination thermal perception module, i.e., the visible light feature credibility weight and the infrared feature credibility weight under the current illumination conditions, and the visible light feature credibility weight and the infrared feature credibility weight under the current thermal conditions, are used to adjust the target category approximation estimate and the target position offset estimate.

2. The infrared-visible light fusion recognition system based on modal difference feature guidance according to claim 1 is characterized in that: The infrared feature extraction unit specifically includes a first base layer, a first dense fusion layer, a second dense fusion layer, a third dense fusion layer and a fourth dense fusion layer; the visible light feature extraction unit specifically includes a second base layer, a fifth dense fusion layer, a sixth dense fusion layer, a seventh dense fusion layer and an eighth dense fusion layer; the feature guidance unit includes a first modal feature difference guidance subunit, a second modal feature difference guidance subunit, a third modal feature difference guidance subunit and a fourth modal feature difference guidance subunit, wherein: The first base layer is used to input an original visible light image and obtain a first-scale visible light feature, the second base layer is used to input an original infrared image and obtain a first-scale infrared feature, and the first modal feature difference guiding subunit is used to fuse the first-scale visible light feature and the first-scale infrared feature, and obtain a first-scale infrared-visible light fusion vector, a first weighted infrared feature, and a first weighted visible light feature; The first dense fusion layer is used to obtain a second-scale visible light feature by using the first weighted infrared feature and the first-scale visible light feature; the fifth dense fusion layer is used to obtain a second-scale infrared feature by using the first weighted visible light feature and the first-scale infrared feature; the second modal feature difference guiding subunit is used to fuse the second-scale visible light feature, the second-scale infrared feature and the first-scale infrared-visible light fusion vector to obtain a second-scale infrared-visible light fusion vector, a second weighted infrared feature and a second weighted visible light feature; The second dense fusion layer is used to obtain a third-scale visible light feature by using the second weighted infrared feature and the second-scale visible light feature; the sixth dense fusion layer is used to obtain a third-scale infrared feature by using the second weighted visible light feature and the second-scale infrared feature; the third modal feature difference guiding subunit is used to fuse the third-scale visible light feature, the third-scale infrared feature and the second-scale infrared-visible light fusion vector to obtain a third-scale infrared-visible light fusion vector, a third weighted infrared feature and a third weighted visible light feature; The third dense fusion layer is used to obtain a fourth-scale visible light feature by using the third weighted infrared feature and the third-scale visible light feature; the seventh dense fusion layer is used to obtain a fourth-scale infrared feature by using the third weighted visible light feature and the third-scale infrared feature; the fourth modal feature difference guiding subunit is used to fuse the fourth-scale visible light feature, the fourth-scale infrared feature and the third-scale infrared-visible light fusion vector to obtain a fourth-scale infrared-visible light fusion vector, a fourth weighted infrared feature and a fourth weighted visible light feature; The fourth dense fusion layer is used to obtain a fifth-scale visible light feature using the fourth weighted infrared feature and the fourth-scale visible light feature; the eighth dense fusion layer is used to obtain a fifth-scale infrared feature using the fourth weighted visible light feature and the fourth-scale infrared feature.

3. The infrared-visible light fusion recognition system based on modal difference feature guidance according to claim 2 is characterized in that: The first base layer and the second base layer are both composed of a 5×5 convolution kernel with a step size of 1, a 1×1 convolution kernel with a step size of 1, and a 3×3 convolution kernel with a step size of 2; the first dense fusion layer, the second dense fusion layer, the third dense fusion layer, the fourth dense fusion layer, the fifth dense fusion layer, the sixth dense fusion layer, the seventh dense fusion layer and the eighth dense fusion layer are all composed of the shullflenetv2 basic convolution kernel with a step size of 2.

4. The infrared-visible light fusion recognition system based on modal difference feature guidance according to claim 1 is characterized in that: The light heat perception unit includes two convolutional layers and three fully connected layers. A RELU activation function and a 2×2 maximum pooling layer are used between the convolutional layers and the fully connected layers to compress and extract features. The light intensity loss of the light heat perception unit is: Among them, w d and w n are the minimum and maximum illumination values ​​of the target area acquired by the illumination and heat sensing unit, and The actual labels for day and night, The infrared intensity loss of the light and heat sensing unit for: Among them, m d and m n are the lowest and highest heat values ​​of the target area acquired by the light heat sensing unit, and The true labels for cold and hot.

5. The infrared-visible light fusion recognition system based on modal difference feature guidance according to claim 2 is characterized in that: The cascade region proposal module includes a feature fusion unit, a classification regression unit and a parameter adjustment unit, wherein: The feature fusion unit is used to fuse the visible light features of the third scale, the visible light features of the fourth scale, the visible light features of the fifth scale, the infrared features of the third scale, the infrared features of the fourth scale, and the infrared features of the fifth scale to obtain a fused image; The classification regression unit is used to perform probability prediction on each target type in the fused image to obtain a target category approximation estimate, and calculate the intersection-over-union ratio of each target position prediction frame and the true value frame to obtain a target position offset estimate; The parameter adjustment unit adjusts the target category approximation estimation value and the target position offset estimation value through the thermal illumination perception parameters output by the illumination thermal perception module, and obtains the category prediction structure and position prediction result of the target in the image.

6. An infrared-visible light fusion recognition method based on modal difference feature guidance, characterized in that: include: S1: constructing an infrared-visible light fusion recognition system guided by modal difference features as described in any one of claims 1 to 5; S2: training the dual-stream backbone network module, the light heat perception module and the cascade region proposal module in the infrared-visible light fusion recognition system to obtain a trained infrared-visible light fusion recognition system; S3: Input the original infrared image and the original visible light image of the same scene into the trained infrared-visible light fusion recognition system to obtain the target type and target position in the image.

7. The infrared-visible light fusion recognition method guided by modality difference features according to claim 6 is characterized in that: The S2 includes: A first training data set is formed by using a large number of pictures with thermal value labels and lighting value labels to train the light thermal perception module to obtain a trained light thermal perception module; The infrared-visible light fusion recognition system is constructed using the original dual-stream backbone network module, the trained light heat perception module and the original cascade region proposal module, and the original dual-stream backbone network module and the original cascade region proposal module in the infrared-visible light fusion recognition system are trained using a second training data set formed by images with target position labels and target type labels to obtain a trained infrared-visible light fusion recognition system.

Images