A method for fusion of infrared and visible light under weak registration and binocular imaging device

Abstract

The present invention discloses a method for fusion of infrared and visible light under weak registration and a binocular imaging device, the method comprising: obtaining a visible light image and an infrared image for the same target, using an image weak registration method to generate a target image pair; inputting the target image pair into a trained deep learning fusion model to obtain an output fusion image; the deep learning fusion model comprises a significant feature extraction module, an encoder, a fusion module and a decoder; in the binocular imaging device, a visible light camera and an infrared camera capture visible light images and infrared images respectively, an AI processor first corrects and crops the infrared image and the visible light image, and then uses a deep learning fusion model to perform fusion processing, and obtains a fusion image to be displayed on a terminal display. The present invention improves imaging quality and allows the fused image pair to have registration deviations, effectively highlights significant information, reduces visual artifacts, and is more conducive to the all-weather operation of the imaging system.

Description

Technical Field

[0001] The present invention relates to the field of image fusion, and in particular to an infrared and visible light fusion method under weak registration and a binocular imaging device. Background Art

[0002] Infrared and visible light multi-sensor imaging systems provide complementary information about the observed area and have been widely used in medical, security, military and other fields to enhance human visual perception, target detection and target recognition. Infrared images are imaged using thermal radiation information, so the target can be easily distinguished from the background. Visible images taken by high-resolution cameras contain rich high-resolution texture details. These two types of images can be fused to provide complementary scene information from different aspects to improve imaging quality.

[0003] In recent years, infrared and visible light fusion imaging technology has been widely used, which has improved the information utilization of multimodal sensors and further extended the working time of imaging systems. However, there are still some disadvantages. First, the thermal radiation characteristics of infrared images and the texture details of visible light images are difficult to balance in the fused image. The fused image is only similar to one source image, resulting in information loss. Secondly, in practical applications, due to the different resolutions of infrared and visible light images, as well as the different viewing angles and alignment methods of the two cameras, it is necessary to register the visible light and infrared images. However, the performance of most fusion methods depends on the registration accuracy. Therefore, the weak registration image fusion method based on infrared and visible light imaging systems is particularly important in the field of image information fusion. Developing an algorithm that can maintain good image fusion performance even when there is registration deviation in the image is very helpful to the application and development of information fusion technology. Summary of the invention

[0004] The purpose of the present invention is to provide a method for fusing infrared and visible light under weak registration and a binocular imaging device, so as to improve the imaging quality and allow the fused image pair to have registration deviation, effectively highlight the significant information, reduce visual artifacts, and be more conducive to the all-weather operation of the imaging system.

[0005] In order to achieve the above tasks, the present invention adopts the following technical solutions:

[0006] A method for fusing infrared and visible light under weak registration, comprising:

[0007] Obtain visible light images and infrared images of the same target, use image weak registration method to correct and crop the infrared image and visible light image pair, and generate target image pair;

[0008] The target image pair is input into a trained deep learning fusion model to obtain an output fusion image; the deep learning fusion model includes a significant feature extraction module, an encoder, a fusion module and a decoder, wherein:

[0009] The salient feature extraction module is used to extract salient features according to the target image pair to obtain an image salient feature pair;

[0010] The input of the encoder is a target image pair and an image salient feature pair; the target image pair and the image salient feature pair are first subjected to shallow feature extraction, and then subjected to deep feature extraction to generate an infrared deep feature map and a visible light deep feature map, an infrared salient feature map and a visible light salient feature map;

[0011] The fusion module includes four fusion layers, each of which uses l1-norm and block-based average operator to fuse feature maps; the first fusion layer of the fusion module first fuses the infrared deep feature map and the visible light deep feature map passed by the encoder to generate a fusion feature f 0 , the second and third fusion layers will f 0 They are fused with the infrared salient feature map and the visible light salient feature map to generate the corresponding feature maps f 1 and f 2 ; Feature map pair (f 1 ,f 2 ) After the fourth fusion layer, the fusion feature map f is generated 3 ;

[0012] The decoder is used to fusion feature map f 3 Reconstruction is performed to obtain the final fused image.

[0013] Furthermore, the salient feature extraction module is used to extract salient features according to the target image pair, including:

[0014] The salient feature extraction module is based on the target image Calculate image salient feature pairs Its mathematical model is as follows:

[0015]

[0016] Furthermore, the encoder is a double-layer network structure, and the target image And the salient image features It is input into the encoder in the form of an image tensor and first passes through a 3×3 convolutional layer to complete shallow feature extraction, and obtains the first feature map pair Compared with the first significant feature map The upper and lower layers of the encoder respectively extract the depth features of the infrared image and the depth features of the visible light image based on the first feature map pair and the first significant feature map pair.

[0017] Furthermore, the upper network includes four layers of 3×3 convolutional layers and linear rectification layers ReLU, which are stacked and skipped from left to right, and finally one layer of 3×3 convolutional layer is used to extract features and reorganize channels; in the upper network, the feature map and It is passed to the upper network for deep feature extraction to generate infrared deep feature map and infrared salient features The continuous stacking and skip connection means that each 3×3 convolutional layer and a linear rectification layer ReLU constitute a feature extraction unit, a total of 4 feature extraction units; the input of the first feature extraction unit is the feature map and The input of the second feature extraction unit is the output of the first feature extraction unit and the feature map and The input of the third feature extraction unit is the output of the second feature extraction unit, the output of the first feature extraction unit, and the feature map and The input of the fourth feature extraction unit is the output of the third feature extraction unit, the output of the second feature extraction unit, the output of the first feature extraction unit, and the feature map and This forms a continuous stacking and jump connection.

[0018] Furthermore, the lower network includes, from left to right, three 3×3 convolutional layers and a linear rectifier layer ReLU consisting of a feature extraction unit, and one 3×3 convolutional layer; in the lower network, the feature map and After being processed by 3 layers of feature extraction units, it is passed to 1 layer of 3×3 convolutional layer to re-adjust the channel and generate a visible light deep feature map and visible light salient features

[0019] Furthermore, the fusion layer processes each feature map in the incoming feature map pair separately to generate a feature map pair with weights, and then generates a fused feature map by superimposing the feature map pairs on each other;

[0020] Among them, the feature map is expressed as is the number of channels in the feature map, i = (1, 2, ..., k) represents the index of the feature map; the activity level of the feature map can be measured by The l1-norm is calculated to represent the initial activity level map A iThe mathematical model of (x,y) is as follows:

[0021] A i (x,y)=||f i 1:M (x,y)|| 1

[0022] Where (x, y) represents the coordinates in the activity level graph, and f i 1:M (x,y) is an M-dimensional vector, which represents the feature map f i m The content at position (x,y) in A i (x,y) is the initial activity level map, ||·|| 1 represents l1-norm calculation;

[0023] The initial activity level map A is processed using a block-based averaging operator. i (x,y) processing to obtain the final activity level map Its mathematical model can be expressed as:

[0024]

[0025] Where r is the size of the operating factor; using the final activity level diagram Calculate the feature map weight w i (x,y):

[0026]

[0027] Among them, w i (x,y) represents the weight at the corresponding feature map (x,y);

[0028] Using the feature map weight w i (x,y) and the corresponding feature map Multiply to get the fused image feature map Its expression is as follows:

[0029]

[0030] Furthermore, the decoder is composed of four short connection layers connected in sequence, each short connection layer includes a 3×3 convolution layer and a linear rectification layer ReLU; a 3×3 convolution layer is set after the last short connection layer to reconstruct the fused image.

[0031] A binocular imaging device includes a visible light camera, an infrared camera, a terminal display and an AI processor, wherein:

[0032] The visible light camera and the infrared camera capture visible light images and infrared images respectively. The AI ​​processor is loaded with the aforementioned deep learning fusion model. The AI ​​processor first corrects and crops the infrared image and the visible light image, and then uses the deep learning fusion model to perform fusion processing to obtain the fused image I f Displayed on the terminal monitor.

[0033] A terminal device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the infrared and visible light fusion method under weak registration are implemented.

[0034] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the infrared and visible light fusion method under weak registration are implemented.

[0035] Compared with the prior art, the present invention has the following technical features:

[0036] The present invention collects infrared and visible light images in real time, and performs weak registration and fusion imaging on the collected target image pairs; in this scheme, the network structure is designed as a double-layer structure to extract the deep feature mapping of infrared and visible images; the salient features and multiple fusion strategies are combined to generate richer fusion feature maps; the fusion network of this scheme reduces the requirements of the fusion algorithm for image registration accuracy. Compared with the existing technology, this scheme takes into account the fault tolerance of the fusion algorithm for image registration, and effectively avoids the appearance of artifacts in the fused image due to image misregistration. The present invention is very conducive to the application research of infrared and visible light fusion and deep learning technology, and it is expected to be widely used in fields such as medical imaging and night monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the process of the present invention.

[0038] Figure 2 Schematic diagram of the deep learning fusion model of the method of the present invention.

[0039] Figure 3 Schematic diagram of a salient feature extraction module of the method of the present invention.

[0040] Figure 4 Schematic diagram of the network encoder of the method of the present invention.

[0041] Figure 5 Schematic diagram of the fusion block architecture of the method of the present invention.

[0042] Figure 6 Schematic diagram of the fusion strategy of the method of the present invention.

[0043] Figure 7 Schematic diagram of a network decoder of the method of the present invention. DETAILED DESCRIPTION

[0044] Aiming at the problem that the infrared and visible light registration accuracy restricts the quality of the fused image in the infrared and visible light fusion technology, the present invention provides an infrared and visible light weak registration deep learning fusion method and a binocular imaging device. The imaging device simultaneously captures infrared and visible light images, and performs weak registration on the captured images, and finally realizes image fusion imaging.

[0045] Referring to the accompanying drawings, a method for fusion of infrared and visible light under weak registration of the present invention comprises the following steps:

[0046] S1, get the size n for the same target ch ×h vis × vis The visible light image with size n ch ×h ir ×v ir The infrared image and the visible light image are corrected and cropped using the weak image registration method to maintain the same resolution and generate a size n ch ×h×w is the target image pair in For infrared images, is the visible light image, n ch is the channel, h is the height, and w is the width.

[0047] Among them, the weak image registration method can adopt the method in the document J. Han, EJPauwels, and P. De Zeeuw, "Visible and infrared image registration in man-made environments employing hybrid visual features," Pattern Recogn. Lett. 34(1), 42-51(2013)., for example.

[0048] S2, inputting the target image pair into the trained deep learning fusion model to obtain an output fusion image F′; the deep learning fusion model includes a salient feature extraction module, an encoder, a fusion module and a decoder, which are specifically described as follows:

[0049] 1. Salient feature extraction module

[0050] The salient feature extraction module is based on the target image Calculate image salient feature pairs Its mathematical model is as follows:

[0051]

[0052]

[0053] like Figure 3 As shown, the salient feature extraction module provided by this solution is used to extract salient feature pairs of images, where Contains features that are more obvious in infrared images than in visible light images. Contains features that are more obvious in visible light images than in infrared images.

[0054] Among them, the image salient features are Will be passed into the encoder for feature extraction to generate infrared salient feature map and visible light salient features At the same time, the target image It is also passed into the encoder for feature extraction to generate infrared deep feature maps and visible light deep feature map

[0055] 2. Encoder

[0056] This scheme provides an encoder structure for extracting image depth feature maps, such as Figure 4 As shown in the figure, the encoder is designed as a two-layer network structure. The upper and lower layers are used to extract the depth features of infrared images and the depth features of visible light images respectively, so that the model can better balance the infrared and visible light image information during the fusion process.

[0057] Target image pair And the salient image features It is input into the encoder in the form of an image tensor and first passes through a 3×3 convolutional layer to complete shallow feature extraction, and obtains the first feature map pair Compared with the first significant feature map

[0058] First infrared signature The first infrared significant feature map The first visible light feature map is passed to the upper layer of the two-layer network And the first visible light salient feature map is passed to the lower layer network. The input makes the model pay more attention to the rich detail features in the image when extracting image features, ignoring relatively weak image features to reduce fusion image artifacts and lower the requirements of the fusion method on the registration accuracy.

[0059] The upper network includes 4 layers of 3×3 convolutional layers and linear rectification layers ReLU from left to right, with continuous stacking and skip connections, and finally 1 layer of 3×3 convolutional layer for extracting features and reorganizing channels; in the upper network, the feature map and It is passed to the upper network for deep feature extraction to generate infrared deep feature map and infrared salient features The continuous stacking and skip connection means that each 3×3 convolutional layer and a linear rectification layer ReLU constitute a feature extraction unit, a total of 4 feature extraction units; the input of the first feature extraction unit is the feature map and The input of the second feature extraction unit is the output of the first feature extraction unit and the feature map and The input of the third feature extraction unit is the output of the second feature extraction unit, the output of the first feature extraction unit, and the feature map and The input of the fourth feature extraction unit is the output of the third feature extraction unit, the output of the second feature extraction unit, the output of the first feature extraction unit, and the feature map and This forms a continuous stacking and jump connection.

[0060] The lower network consists of three 3×3 convolutional layers and a linear rectifier ReLU feature extraction unit from left to right, and one 3×3 convolutional layer; in the lower network, the feature map and After being processed by 3 layers of feature extraction units, it is passed to 1 layer of 3×3 convolutional layer to re-adjust the channel and generate a visible light deep feature map and visible light salient features

[0061] 3. Fusion Module

[0062] The fusion module provided in this implementation contains four fusion layers, such as Figure 5 As shown; each fusion layer uses l1-norm and block-based average operator to fuse feature maps.

[0063] In this scheme, the first fusion layer of the fusion module first fuses the infrared deep feature map and the visible light deep feature map transmitted by the encoder. Generate fusion feature f 0 , the second and third fusion layers will f 0 Respectively with infrared salient feature map and visible light salient features Fusion is performed to generate corresponding feature maps f 1 and f2 ; Feature map pair (f 1 ,f 2 ) After the fourth fusion layer, the fusion feature map f is generated 3 The multiple fusion strategy uses salient features to balance the infrared thermal radiation characteristics and visible light detail textures, making the fused feature map have richer features.

[0064] Figure 6 The fusion strategy in each fusion layer is shown. The fusion layer processes each feature map in the incoming feature map pair separately to generate a weighted feature map pair, and then generates a fused feature map by superimposing the feature map pairs on each other.

[0065] Among them, the feature map is expressed as is the number of channels in the feature map, i = (1, 2, ..., k) represents the index of the feature map; in this embodiment, k = 2, M = 64. The activity level of the feature map can be measured by f i 1:M The initial activity level map A is represented by the l1-norm calculation of (x,y). i The mathematical model of (x,y) is as follows:

[0066] A i (x,y)=||f i 1:M (x,y)|| 1

[0067] Where (x, y) represents the coordinates in the activity level graph, and f i 1:M (x,y) is an M-dimensional vector, which represents the feature map f i m The content at position (x,y) in A i (x,y) is the initial activity level map, ||·|| 1 represents the l1-norm calculation; this scheme uses a block-based average operator to calculate the initial activity level map A i (x,y) processing to obtain the final activity level map Its mathematical model can be expressed as:

[0068]

[0069] Where r is the size of the operation factor; in this embodiment, the value of r is 1. Using the final activity level map Feature map weight w i (x,y) can be calculated:

[0070]

[0071] Among them, w i (x, y) represents the weight at the corresponding feature map (x, y); this scheme uses the feature map weight w i (x,y) and the corresponding feature map f i m Multiply to get the fused image feature map Its expression is as follows:

[0072]

[0073] The fusion feature map f generated by the fusion module 3 It will be used as the input of the decoder, and the decoder will reconstruct the fused image.

[0074] 4. Decoder

[0075] The decoder consists of four short-connected layers connected in sequence, such as Figure 7 As shown in the figure, each short connection layer includes a 3×3 convolution layer and a linear rectification layer ReLU; a 3×3 convolution layer is set after the last short connection layer to reconstruct the fused image.

[0076] In this scheme, the fusion feature map f 3 After being passed into the decoder, it passes through a series of convolutional layers and linear rectification layers to adjust the channels, and is finally input into a 3×3 convolutional layer to complete the fused image reconstruction and obtain the fused image F′.

[0077] 5. Training of Deep Learning Fusion Model

[0078] In the model training phase of this scheme, the salient feature extraction module and the fusion module are removed from the network to train an autoencoder network model; the encoder in the network can extract multi-scale deep features, and the decoder can reconstruct the input image based on the deep features. During the training process, the training data set is a high-definition image data set, and the Adam function is used to optimize the loss function L(Θ), which is defined as:

[0079] L(θ)=L pixel +λL ssim

[0080] In the above formula, λ is L ssim The weight in the loss function, L pixel is the pixel loss function, which can be expressed as follows:

[0081] L pixel (o,i)=||oi|| 2

[0082] Where i is the infrared image and visible light image input by the model, o is the image reconstructed by the model after the input image i is input; L ssimis the cost function for image similarity, which is defined as:

[0083] L ssim (o,i)=1-SSIM(o,i)

[0084] Where SSIM(·) is the image similarity function, and its mathematical expression is:

[0085]

[0086] In the above formula, μ x , μ y , σ x , σ y , and σ xy are the mean of image x, the mean of image y, the variance of image x, the variance of image y, and the covariance of images x and y. 1 , C 2 and C 3 are all constants to avoid system errors caused by denominators being zero. 1 =(K 1 ×L) 2 , C 1 =(K 1 ×L) 2 , C 3 =C 2 / 2, where K 1 and K 2 is a constant, L is the number of bits per pixel in the image; in one embodiment of the present invention, K 1 =0.01, K 2 =0.03, L = 255. The cost function requires fitting Θ as the deep learning network parameter, λ as the weight control parameter of the similarity cost function; after N times of training, the optimized parameter Θ′ can be obtained.

[0087] On the basis of the above technical solution, the present invention further provides a binocular imaging device, including a visible light camera (resolution 530×354), an infrared camera (resolution 320×240), a terminal display and an AI processor, wherein:

[0088] The visible light camera and the infrared camera capture visible light (450-620nm) images and infrared (8-12μm) images respectively. The AI ​​processor is loaded with the aforementioned deep learning fusion model. The AI ​​processor first corrects and crops the infrared image and the visible light image to maintain the same resolution, and then uses the deep learning fusion model to perform fusion processing to obtain the fused image I f Displayed on the terminal monitor.

[0089] An embodiment of the present application further provides a terminal device, which may be a computer or a server; it includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above-mentioned method for fusion of infrared and visible light under weak registration, for example, the aforementioned S1 to S2.

[0090] The computer program may also be divided into one or more modules / units, one or more modules / units are stored in the memory and executed by the processor to complete the present application. One or more modules / units may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0091] The implementation of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the infrared and visible light fusion method under weak registration are implemented, for example, the aforementioned S1 to S2.

[0092] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0093] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for fusion of infrared and visible light under weak registration, characterized in that: include: Obtain visible light images and infrared images of the same target, use image weak registration method to correct and crop the infrared image and visible light image pair, and generate target image pair; The target image pair is input into a trained deep learning fusion model to obtain an output fusion image; the deep learning fusion model includes a significant feature extraction module, an encoder, a fusion module and a decoder, wherein: The salient feature extraction module is used to extract salient features according to the target image pair to obtain an image salient feature pair; The input of the encoder is a target image pair and an image salient feature pair; the target image pair and the image salient feature pair are first subjected to shallow feature extraction, and then subjected to deep feature extraction to generate an infrared deep feature map and a visible light deep feature map, an infrared salient feature map and a visible light salient feature map; The fusion module includes four fusion layers, each of which uses l1-norm and block-based average operator to fuse feature maps; the first fusion layer of the fusion module first fuses the infrared deep feature map and the visible light deep feature map passed by the encoder to generate a fusion feature f 0 , the second and third fusion layers will f 0 They are fused with the infrared salient feature map and the visible light salient feature map to generate the corresponding feature maps f 1 and f 2 ; Feature map pair (f 1 ,f 2 ) After the fourth fusion layer, the fusion feature map f is generated 3 ; The decoder is used to fusion feature map f 3 Reconstruction is performed to obtain the final fused image.

2. The method for fusion of infrared and visible light under weak registration according to claim 1, characterized in that: The salient feature extraction module is used to extract salient features according to the target image pair, including: The salient feature extraction module is based on the target image Calculate image salient feature pairs Its mathematical model is as follows:

3. The method for fusion of infrared and visible light under weak registration according to claim 1, characterized in that: The encoder is a two-layer network structure. And the salient image features It is input into the encoder in the form of an image tensor and first passes through a 3×3 convolutional layer to complete shallow feature extraction, and obtains the first feature map pair Compared with the first significant feature map The upper and lower layers of the encoder respectively extract the depth features of the infrared image and the depth features of the visible light image based on the first feature map pair and the first significant feature map pair.

4. The method for fusion of infrared and visible light under weak registration according to claim 3, characterized in that: The upper network consists of 4 layers of 3×3 convolutional layers and linear rectification layers ReLU, which are stacked and skipped from left to right, and finally 1 layer of 3×3 convolutional layer is used to extract features and reorganize channels; In the upper network, the feature map and It is passed to the upper network for deep feature extraction to generate infrared deep feature map and infrared salient features The continuous stacking and skip connection means that each 3×3 convolutional layer and a linear rectification layer ReLU constitute a feature extraction unit, a total of 4 feature extraction units; The input of the first feature extraction unit is the feature map and The input of the second feature extraction unit is the output of the first feature extraction unit and the feature map and The input of the third feature extraction unit is the output of the second feature extraction unit, the output of the first feature extraction unit, and the feature map and The input of the fourth feature extraction unit is the output of the third feature extraction unit, the output of the second feature extraction unit, the output of the first feature extraction unit, and the feature map and This forms a continuous stacking and jump connection.

5. The method for fusion of infrared and visible light under weak registration according to claim 3, characterized in that: The lower network consists of three 3×3 convolutional layers and a linear rectifier ReLU feature extraction unit from left to right, and one 3×3 convolutional layer; in the lower network, the feature map and After being processed by 3 layers of feature extraction units, it is passed to 1 layer of 3×3 convolutional layer to re-adjust the channel and generate a visible light deep feature map and visible light salient features 6. The method for fusion of infrared and visible light under weak registration according to claim 1, characterized in that: The fusion layer processes each feature map in the incoming feature map pair separately to generate a weighted feature map pair, and then generates a fused feature map by superimposing the feature map pairs on each other; Among them, the feature map is represented as f i m , m∈{1,2,…,M} is the number of channels in the feature map, i=(1,2,…,k) represents the index of the feature map; the activity level of the feature map can be measured by f i 1:M The initial activity level map A is represented by the l1-norm calculation of (x,y). i The mathematical model of (x,y) is as follows: A i (x,y)=||f i 1:M (x,y)||1 Where (x, y) represents the coordinates in the activity level graph, and f i 1:M (x,y) is an M-dimensional vector, which represents the feature map f i m The content at position (x,y) in A i (x,y) is the initial activity level map, ||·||1 indicates the l1-norm calculation; The initial activity level map A is processed using a block-based averaging operator. i (x,y) processing to obtain the final activity level map Its mathematical model can be expressed as: Where r is the size of the operating factor; using the final activity level diagram Calculate the feature map weight w i (x,y): Among them, w i (x,y) represents the weight at the corresponding feature map (x,y); Using the feature map weight w i (x,y) and the corresponding feature map f i m Multiply to get the fused image feature map Its expression is as follows:

7. The method for fusion of infrared and visible light under weak registration according to claim 1, characterized in that: The decoder is composed of four short connection layers connected in sequence, each short connection layer includes a 3×3 convolution layer and a linear rectification layer ReLU; a 3×3 convolution layer is set after the last short connection layer to reconstruct the fused image.

8. A binocular imaging device, characterized in that: It includes visible light camera, infrared camera, terminal display and AI processor, among which: The visible light camera and the infrared camera capture visible light images and infrared images respectively, and the AI ​​processor is loaded with the deep learning fusion model in the infrared and visible light fusion method under weak registration according to claim 1; the AI ​​processor first corrects and crops the infrared image and the visible light image, and then uses the deep learning fusion model to perform fusion processing to obtain a fused image I f Displayed on the terminal monitor.

9. A terminal device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for fusion of infrared and visible light under weak registration according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for fusion of infrared and visible light under weak registration according to any one of claims 1 to 7 is implemented.

Images