Dual light fusion method, device and equipment based on multi-scale region attention mechanism

Abstract

The application discloses a dual-light fusion method and device based on a multi-scale region attention mechanism and equipment, the method uses an encoding network to extract features of infrared and visible light images respectively, uses a classification network to extract discriminative features of the infrared image and the visible light image respectively, and sends the extracted features to a fusion module by the encoding network and the classification network to obtain fused features, and finally sends the fused features to a decoding network to obtain a reconstructed fusion image. In order to improve the fusion effect and accelerate the fusion speed, the application designs a fusion strategy based on a multi-scale region attention mechanism, and has certain practical value for display of the fusion image and subsequent all-weather image perception tasks.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a dual-light fusion method, apparatus, and device based on a multi-scale regional attention mechanism. Background Technology

[0002] Due to hardware limitations in imaging equipment, a single type of sensor cannot comprehensively characterize an imaging scene. For example, visible light images typically contain rich textural details, but are easily affected by weather and lighting conditions, resulting in the loss of targets within the scene. Conversely, infrared images characterize the thermal radiation emitted by objects, effectively highlighting prominent targets such as pedestrians and vehicles, but lack detailed descriptions of the scene. Dual-light fusion technology aims to fuse these complementary pieces of information into a single image.

[0003] Traditional image fusion algorithms typically measure the activity level of images in the spatial or transform domain and manually design fusion rules to achieve image fusion. Traditional image fusion frameworks are mainly based on multi-scale transformations, sparse representations, subspaces, saliency, and variational models. While these traditional image fusion algorithms have achieved some results, several challenges remain hindering their further development. In recent years, deep learning, with its powerful feature extraction and representation capabilities, has dominated the development of computer vision and is also showing unstoppable momentum in the field of image fusion. Based on the different network architectures used, deep learning-based fusion algorithms can be divided into three categories: those based on autoencoders (AE), convolutional neural networks (CNN), and generative adversarial networks (GAN).

[0004] Autoencoder-based image fusion frameworks typically train an autoencoder on a large-scale natural image dataset for feature extraction and image reconstruction. Then, a manually designed fusion strategy is used to fuse the deep features extracted by the encoding network as input to the decoding network. Some researchers are also dedicated to designing more complex and interpretable fusion strategies to improve fusion performance.

[0005] However, the aforementioned algorithms generate pixel-level saliency maps the same size as the image and perform importance assessments, which is very time-consuming. Furthermore, people are often more attracted to areas with strong gradient and texture information in an image, paying less attention to smooth background areas. A pixel-level saliency map of the entire image can introduce unnecessary noise, affecting the quality of the fused image. Summary of the Invention

[0006] The purpose of this invention is to overcome at least one defect in the prior art and to provide a dual-light fusion method, apparatus and device based on a multi-scale regional attention mechanism. It aims to focus more on the discriminative complementary regions of infrared and visible light during fusion, thereby improving the fusion speed while reducing background noise.

[0007] The technical solution of this invention is implemented as follows: This invention discloses a dual-light fusion method based on a multi-scale region attention mechanism, comprising the following steps:

[0008] S1) Acquire a visible light image and its registered infrared image, and input both the visible light image and the infrared image into the coding network and the classification network;

[0009] S2) The encoding network extracts features from the visible light image and the infrared image respectively, and outputs the first visible light feature map and the first infrared feature map; the classification network extracts discriminative features from the infrared image and the visible light image respectively, and outputs the second visible light feature map and the second infrared feature map.

[0010] S3) After fusing the first visible light feature map, the first infrared feature map, the second visible light feature map, and the second infrared feature map to obtain the final fused feature map, the result is output to the decoding network;

[0011] The S4 decoding network reconstructs the image using the final fused feature map to obtain the fused image.

[0012] Furthermore, before step S1), there is also a step of constructing and training an autoencoder. The autoencoder includes an encoding network and a decoding network. The encoding network includes multiple encoding modules connected in sequence. The first and last encoding modules in the encoding network each include a convolutional layer and an activation layer connected in sequence. The remaining encoding modules in the encoding network each include a convolutional layer, an activation layer, and a pooling layer connected in sequence. The decoding network includes multiple decoding modules connected in sequence. The first and last decoding modules in the decoding network each include a convolutional layer and an activation layer connected in sequence. The remaining decoding modules in the decoding network each include a deconvolutional layer and an activation layer connected in sequence.

[0013] Furthermore, dense connections are introduced into the coding network, specifically by connecting the output of each coding module with a pooling layer to the input of all subsequent coding modules.

[0014] Furthermore, cross-layer connections are added between the encoding network and the decoding network, specifically including connecting the output of each encoding module with a pooling layer in the encoding network to the input of the corresponding decoding module in the decoding network.

[0015] Furthermore, the loss function of the autoencoder is:

[0016] L = L pixel +γLgradient

[0017] L pixel =||XY||2

[0018]

[0019] Among them, L pixel For pixel distribution, L gradient Let γ be the gradient distribution, γ be the loss balance factor, and X and Y represent the input image and the predicted image, respectively. and These represent the gradients calculated with respect to the input image and the predicted image, respectively.

[0020] Further, step S3) specifically includes: obtaining multiple second visible light feature maps and multiple second infrared feature maps output by multiple convolutional modules in the classification network; taking one of the multiple convolutional modules in the classification network as the main convolutional module and taking the remaining one or more convolutional modules as auxiliary convolutional modules; processing the second visible light feature map and the second infrared feature map respectively to obtain a visible light probability map and an infrared probability map; performing a thresholding operation on each visible light probability map and infrared probability map respectively to obtain a visible light region probability map and an infrared region probability map; performing a dot product operation on each of the multiple visible light region probability maps with the first visible light feature map to obtain multiple visible light weighted feature maps; performing a dot product operation on each of the multiple infrared region probability maps with the first infrared feature map respectively to obtain multiple infrared weighted feature maps; performing an addition operation on each visible light weighted feature map with the corresponding infrared weighted feature map to obtain multiple fused feature maps; and merging the multiple fused feature maps to obtain the final fused feature map.

[0021] The process of merging multiple fusion feature maps involves: for key information regions, averaging the fusion features of multiple fusion feature maps to obtain the final fusion result; for non-key information regions, using the fusion features of the fusion feature map corresponding to the main convolution module as the final fusion result.

[0022] Furthermore, when performing thresholding operations on the visible light probability map and infrared probability map corresponding to the main convolution module, all probabilities greater than the threshold t are retained, and probabilities less than the threshold t are set to t. When performing thresholding operations on the visible light probability map and infrared probability map corresponding to the auxiliary convolution module, only all probabilities greater than the threshold t are retained, and probabilities less than the threshold t are set to 0.

[0023] Furthermore, multiple fusion feature maps are merged, specifically including: for the portion of the fusion feature map corresponding to the main convolution module that is above the threshold t, the average value of the fusion features of multiple fusion feature maps is calculated as the fusion result of the final fusion feature map; for the portion of the fusion feature map corresponding to the main convolution module that is below the threshold t, the fusion features of the fusion feature map corresponding to the main convolution module are used as the fusion result of the final fusion feature map.

[0024] Furthermore, the last convolutional module in the classification network is taken as the main convolutional module, and the remaining one or more convolutional modules are taken as auxiliary convolutional modules.

[0025] Further, the second visible light feature map and the second infrared feature map are processed, specifically including: upsampling the second visible light feature map and the second infrared feature map to the same size as the visible light image and the infrared image registered with it, respectively, and performing exponential normalization to obtain the visible light probability map and the infrared probability map.

[0026] This invention also discloses a dual-light fusion device based on a multi-scale region attention mechanism, comprising:

[0027] The encoding network module is used to receive a visible light image and an infrared image registered with it, extract features from the visible light image and the infrared image respectively, and output a first visible light feature map and a first infrared feature map to the fusion module.

[0028] The classification network module is used to receive a visible light image and an infrared image registered with it, extract features from the infrared image and the visible light image respectively, and output a second visible light feature map and a second infrared feature map to the fusion module.

[0029] The fusion module is used to receive the first visible light feature map and the first infrared feature map output by the encoding network module, and the second visible light feature map and the second infrared feature map output by the classification network module, and perform a fusion operation to obtain a fused feature map and output it to the decoding network module.

[0030] The decoding network module is used to reconstruct the image from the fused feature map to obtain the fused image.

[0031] This invention discloses a dual-light fusion device based on a multi-scale regional attention mechanism, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the dual-light fusion method described above.

[0032] The present invention has at least the following beneficial effects:

[0033] To improve the fusion effect and accelerate the fusion speed, this invention designs a fusion strategy based on a multi-scale region attention mechanism. By improving the feature fusion strategy, the dual-light fusion effect is enhanced. The region-based attention mechanism not only ensures the fusion effect but also accelerates the fusion process. It has high practical value for the visualization of dual-light fused images and subsequent all-weather image perception tasks.

[0034] To enhance the diversity of image features extracted by autoencoders, this invention introduces pooling layers and deconvolution layers into both the encoding and decoding networks. Furthermore, to prevent pixel loss, this invention incorporates dense connections and cross-layer connections into both the encoding and decoding networks.

[0035] This invention trains a classifier to obtain interpretable region probability maps with deep features. Unlike the feature extraction capabilities of encoding networks, the classification network captures the most significant features that can distinguish between infrared and visible light images. The fusion method of this invention can extract complementary information more efficiently and can directly output the multi-scale features of the classifier, thereby improving the fusion speed. In addition, the method described in this invention is universal for fusion of different modal images (medical images, multi-exposure images, multi-focus images). Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 A flowchart illustrating the dual-light fusion method based on a multi-scale region attention mechanism provided in an embodiment of the present invention;

[0038] Figure 2 The training flowchart of the dual-light fusion method based on multi-scale region attention mechanism provided in the embodiments of the present invention is shown below.

[0039] Figure 3 A test flowchart of the dual-light fusion method based on a multi-scale region attention mechanism provided in an embodiment of the present invention;

[0040] Figure 4 This is a schematic diagram of the autoencoder network structure provided in an embodiment of the present invention;

[0041] Figure 5 A flowchart illustrating the fusion strategy provided in an embodiment of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature; in the description of this invention, unless otherwise stated, "a plurality of" or "several" means two or more.

[0044] See Figures 1 to 5 This invention provides a dual-light fusion method based on a multi-scale region attention mechanism, comprising the following steps:

[0045] S1) Acquire a visible light image and its registered infrared image, and input both the visible light image and the infrared image into the trained encoding network and the trained classification network;

[0046] S2) The encoding network extracts features from the visible light image and the infrared image respectively, and outputs the first visible light feature map and the first infrared feature map; the classification network extracts features from the infrared image and the visible light image respectively, and outputs the second visible light feature map and the second infrared feature map.

[0047] S3) After fusing the first visible light feature map, the first infrared feature map, the second visible light feature map, and the second infrared feature map to obtain the final fused feature map, the result is output to the trained decoding network.

[0048] The S4 decoding network reconstructs the image using the final fused feature map to obtain the fused image.

[0049] Generally, the acquired infrared images are single-channel images. Therefore, before inputting the acquired RGB visible light images into the encoding network and classification network, a single-channel visible light image needs to be generated first, registered with the single-channel infrared image, and a single-channel visible light image needs to be generated. Specifically, this includes converting the RGB visible light image to YCrCb format and extracting the Y channel.

[0050] Furthermore, step S1) is preceded by a step of constructing and training an autoencoder. The autoencoder includes an encoding network and a decoding network. To enable the autoencoder to effectively generate visible light and infrared images, this invention uses a pre-registered infrared-visible light dataset to train the autoencoder. In this embodiment, the pre-registered infrared-visible light dataset has 4000 training pairs and 300 test pairs, with the vast majority of images being 1024*768*3 pixels in size. The number of channels in the infrared and visible light images maintains a one-to-one correspondence. For example, for an RGB visible light image, it can first be converted to YCrCb format and the Y channel can be extracted to generate a single-channel image; for a three-channel infrared image, its first channel can be extracted.

[0051] Considering GPU memory, this invention first cuts it into 256*256 squares without overlap. When training the autoencoder, the input to the autoencoder network and the classification network is an infrared or visible light square of size 256*256.

[0052] Furthermore, after obtaining the fused image, the autoencoder network of the present invention adds the corresponding color information stored in the CrCb channel and converts it into a fused image in RGB format.

[0053] like Figure 2 As shown, in this invention, only the encoding and decoding networks need to be trained, and a classification network is trained as the input to the fusion module. The fusion module is added directly after the encoding network during the testing phase to obtain fused features as the input to the decoding network.

[0054] Furthermore, to enhance the diversity of image features extracted by the autoencoder network, this invention introduces pooling layers and deconvolution layers in both the encoding and decoding networks. Simultaneously, to avoid pixel loss, this invention introduces dense connections and cross-layer connections in both the encoding and decoding networks.

[0055] Furthermore, the encoding network includes multiple sequentially connected encoding modules. The head and tail encoding modules within the encoding network each include sequentially connected convolutional layers and activation layers. The remaining encoding modules within the encoding network each include sequentially connected convolutional layers, activation layers, and pooling layers. Similarly, the decoding network includes multiple sequentially connected decoding modules. The head and tail decoding modules within the decoding network each include sequentially connected convolutional layers and activation layers. The remaining decoding modules within the decoding network each include sequentially connected deconvolutional layers and activation layers. The encoding and decoding networks are symmetrically arranged.

[0056] Furthermore, dense connections are introduced into the coding network, specifically including: the output of each coding module with a pooling layer in the coding network is the input of all subsequent coding modules.

[0057] Furthermore, cross-layer connections are added between the encoding network and the decoding network, specifically including: the output of each encoding module with a pooling layer in the encoding network is connected to the input of the corresponding decoding module in the decoding network.

[0058] In the encoding network, multiple encoding modules with pooling layers correspond one-to-one with multiple decoding modules with deconvolution layers in the decoding network. The outputs of the multiple encoding modules with pooling layers in the encoding network are connected one-to-one with the inputs of the multiple decoding modules with deconvolution layers in the decoding network. For example, in this embodiment, the output of the first encoding module with a pooling layer in the encoding network is connected to the input of the last decoding module with a deconvolution layer in the decoding network; the output of the second encoding module with a pooling layer in the encoding network is connected to the input of the second-to-last decoding module with a deconvolution layer in the decoding network; the output of the last encoding module with a pooling layer in the encoding network is connected to the input of the first decoding module with a deconvolution layer in the decoding network, and so on.

[0059] The network structure of the entire autoencoder in one embodiment is as follows: Figure 4 As shown, the encoding network consists of five encoding modules, and the decoding network consists of five decoding modules. Pooling layers are added to the second, third, and fourth encoding modules of the encoding network, and dense connections are added to the second, third, fourth, and fifth encoding modules. Cross-layer connections are added between the second, third, and fourth modules of the encoding network and the decoding network.

[0060] Furthermore, to ensure that the autoencoder can correctly generate images and that the pixel distribution and gradient distribution of the original image and the generated image are the same, the loss function of the autoencoder in this invention uses the Euclidean distance of pixels plus the Euclidean distance of gradients between the original image and the predicted image, as shown in the following formula:

[0061] L = L pixel +γL gradient

[0062] L pixel =||XY||2

[0063]

[0064] Among them, L pixel For pixel distribution, L gradient The gradient distribution is represented by γ, which is the loss balancing factor. γ can be chosen as needed, such as 0.5. X and Y represent the input image and the predicted image, respectively. and These represent the gradients calculated with respect to the input image and the predicted image, respectively.

[0065] Further, see Figure 5After receiving a visible light image, the classification network arbitrarily selects multiple convolutional modules (generally, the selected feature sizes will be different) to output multiple second visible light feature maps respectively. After receiving an infrared image, the multiple convolutional modules of the classification network output multiple second infrared feature maps respectively.

[0066] Step S3) specifically includes: obtaining multiple second visible light feature maps and multiple second infrared feature maps output by multiple convolutional modules in the classification network; taking one of the convolutional modules in the classification network as the main convolutional module and the remaining one or more convolutional modules as auxiliary convolutional modules; processing the second visible light feature maps and the second infrared feature maps respectively to obtain visible light probability maps and infrared probability maps; performing thresholding operations on each visible light probability map and infrared probability map respectively to obtain visible light region probability maps and infrared region probability maps; performing matrix dot multiplication operations on multiple visible light region probability maps with the first visible light feature map respectively to obtain multiple visible light weighted feature maps; performing matrix dot multiplication operations on multiple infrared region probability maps with the first infrared feature map respectively to obtain multiple infrared weighted feature maps; performing addition operations on each visible light weighted feature map and the corresponding infrared weighted feature map (the visible light weighted feature map and the infrared weighted feature map corresponding to the same convolutional module in the classification network correspond to each other) to obtain multiple fused feature maps; and merging the multiple fused feature maps to obtain the final fused feature map.

[0067] Multiple fusion feature maps are merged, specifically: for the portion of the fusion feature map corresponding to the main convolution module that falls above a threshold t (i.e., the key information portion), the average of the fusion features from multiple fusion feature maps is calculated as the final fusion result. For the portion of the fusion feature map corresponding to the main convolution module that falls below a threshold t (i.e., the non-key information portion), the fusion features from the fusion feature map corresponding to the main convolution module are directly used as the final fusion result. The portion equal to the threshold t can be classified as either the key information portion or the non-key information portion. That is, for the portion equal to the threshold t, either the average of the fusion features from multiple fusion feature maps is used as the final fusion result, or the fusion features from the fusion feature map corresponding to the main convolution module are directly used as the final fusion result.

[0068] Furthermore, when performing thresholding operations on the visible light probability map and infrared probability map corresponding to the main convolution module, all probabilities greater than the threshold t are retained, and probabilities less than the threshold t are directly set to t. The formula is as follows: For the visible light probability map and infrared probability map corresponding to the auxiliary convolution module, when performing a thresholding operation, only all probabilities greater than the threshold t are retained, and probabilities less than the threshold t are directly set to 0. The threshold t is determined based on experience and experiments. The selection of the threshold t needs to be able to distinguish between those with discriminative features and those without. The thresholding is used for region segmentation, and after the thresholding operation, the probability map can be divided into two regions. This invention uses the thresholding operation to reduce background noise and ensure that the weights within the background region remain consistent.

[0069] Furthermore, the last convolutional module in the classification network is taken as the main convolutional module, and the remaining one or more convolutional modules are taken as auxiliary convolutional modules.

[0070] Further processing is performed on the second visible light feature map and the second infrared feature map, specifically including: upsampling the second visible light feature map and the second infrared feature map to the same size as the visible light image and the infrared image registered with it, respectively, and then performing exponential normalization to obtain the visible light probability map and the infrared probability map. Upsampling generally uses, but is not limited to, linear interpolation. Exponential normalization is used to normalize the target to the range [0,1].

[0071] The encoding and decoding networks of this invention are used for image generation, and the classification network is used for binary classification of infrared and visible light. The fusion module does not participate in training.

[0072] Considering that deep learning extracts different image features for different tasks, this invention incorporates a classification network to extract discriminative information from infrared and visible light images (i.e., information that can distinguish between infrared and visible light images), which is also complementary information between the two. Figure 2 As shown, the classification network will ultimately output the probability that the input image belongs to an infrared image (category 0) or a visible light image (category 1). On the other hand, the features extracted by shallow networks and deep networks are also different. This invention will utilize the multi-scale features of the classification network for information filtering.

[0073] The classification network of this invention can be ResNet, VGG, DenseNet or SENet, etc. The classification network in this embodiment is ResNet.

[0074] One specific implementation involves using the second visible light feature map and the second infrared feature map output from the first three convolutional modules (convolutional layer, activation layer, and pooling layer) of the ResNet network. These are upsampled to the same size as the image and exponentially normalized to obtain a probability map. The last convolutional module in the first three convolutional modules of the ResNet network is used as the main convolutional module, and the first two convolutional modules are used as auxiliary convolutional modules. The image output from the last convolutional module has richer details and can be used as the main image, while the images output from the other auxiliary convolutional modules are used as auxiliary images. A thresholding operation is performed on the main image and the two auxiliary images to obtain three region probability maps, including one main region probability map and two auxiliary region probability maps. Specifically, for the main region probability map, a thresholding operation is performed, retaining all probabilities greater than a threshold t and directly setting probabilities less than a threshold t to t. For the auxiliary region probability maps, only all probabilities greater than a threshold t are retained, and probabilities less than a threshold t are directly set to 0.

[0075] Based on the same inventive concept, the present invention also provides a dual-light fusion device based on a multi-scale region attention mechanism, comprising:

[0076] The encoding network module is used to receive a visible light image and an infrared image registered with it, extract features from the visible light image and the infrared image respectively, and output a first visible light feature map and a first infrared feature map to the fusion module.

[0077] The classification network module is used to receive a visible light image and an infrared image registered with it, extract discriminative features from the infrared image and the visible light image respectively, and output a second visible light feature map and a second infrared feature map to the fusion module.

[0078] The fusion module is used to receive the first visible light feature map and the first infrared feature map output by the encoding network module, and the second visible light feature map and the second infrared feature map output by the classification network module, and perform a fusion operation to obtain a fused feature map and output it to the decoding network module.

[0079] The decoding network module is used to reconstruct the image from the fused feature map to obtain the fused image.

[0080] Based on the same inventive concept, the present invention also provides a dual-light fusion device based on a multi-scale regional attention mechanism, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the dual-light fusion method described above.

[0081] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is typically used to control the overall operation of the electronic device. In this embodiment, the processor is used to run program code stored in the memory or process data, such as program code for a dual-light fusion method based on a multi-scale region attention mechanism.

[0082] The memory includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of the electronic device, such as the hard disk or memory of the electronic device. In other embodiments, the memory may also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Of course, the memory may include both internal storage units and external storage devices of the electronic device. In this embodiment, the memory is typically used to store operating methods and various application software installed on the electronic device, such as program code for a dual-light fusion method based on a multi-scale region attention mechanism. Furthermore, the memory can also be used to temporarily store various types of data that have been output or will be output.

[0083] Based on the same inventive concept, the present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the dual-light fusion method described above.

[0084] This invention belongs to the fields of image processing and deep learning, and particularly relates to the field of assisted driving. It is a method for extracting complementary information from infrared and visible light images and generating fused images. The key point of this invention is to improve the dual-light fusion effect by improving the feature fusion strategy. It adopts a region-based attention mechanism to accelerate the fusion process while ensuring the fusion effect. To improve the fusion effect and speed, a fusion strategy based on a multi-scale region attention mechanism is designed, which has high practical value for the visualization of dual-light fused images and subsequent all-weather image perception tasks.

[0085] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0086] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0087] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0088] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0089] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0090] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dual-light fusion method based on a multi-scale region attention mechanism, characterized in that, Includes the following steps: S1) Acquire a visible light image and an infrared image registered with it, and input both the visible light image and the infrared image into the encoding network and the classification network; the encoding network includes multiple encoding modules connected in sequence, and dense connections are introduced into the encoding network, specifically including: connecting the output of each encoding module with a pooling layer in the encoding network to the input of all subsequent encoding modules; S2) The encoding network extracts features from the visible light image and the infrared image respectively, and outputs the first visible light feature map and the first infrared feature map; the classification network extracts features from the infrared image and the visible light image respectively, and outputs the second visible light feature map and the second infrared feature map. S3) After fusing the first visible light feature map, the first infrared feature map, the second visible light feature map, and the second infrared feature map to obtain the final fused feature map, the result is output to the decoding network. The decoding network includes multiple decoding modules connected in sequence. Cross-layer connections are added between the encoding network and the decoding network. Specifically, the output of each encoding module with a pooling layer in the encoding network is connected to the input of the corresponding decoding module in the decoding network. S4) The decoding network reconstructs the image using the final fused feature map to obtain the fused image.

2. The dual-light fusion method based on a multi-scale region attention mechanism as described in claim 1, characterized in that: Before step S1), there is also a step of building and training an autoencoder. The autoencoder includes an encoding network and a decoding network. The encoding modules at the beginning and end of the encoding network each include a convolutional layer and an activation layer connected in sequence. The remaining encoding modules in the encoding network each include a convolutional layer, an activation layer, and a pooling layer connected in sequence. The decoding modules at the beginning and end of the decoding network each include a convolutional layer and an activation layer connected in sequence. The remaining decoding modules in the decoding network each include a deconvolutional layer and an activation layer connected in sequence.

3. The dual-light fusion method based on a multi-scale region attention mechanism as described in claim 2, characterized in that: The loss function of the autoencoder is: ； in, It is the loss balance factor. and These represent the input image and the predicted image, respectively. and These represent the gradients calculated with respect to the input image and the predicted image, respectively.

4. The dual-light fusion method based on a multi-scale region attention mechanism as described in claim 1, characterized in that: Step S3) specifically includes: obtaining multiple second visible light feature maps and multiple second infrared feature maps output by multiple convolutional modules in the classification network; taking one of the multiple convolutional modules in the classification network as the main convolutional module and the remaining one or more convolutional modules as auxiliary convolutional modules; processing the second visible light feature maps and the second infrared feature maps respectively to obtain visible light probability maps and infrared probability maps; performing thresholding operations on each visible light probability map and infrared probability map respectively to obtain visible light region probability maps and infrared region probability maps; performing dot product operations on multiple visible light region probability maps with the first visible light feature map respectively to obtain multiple visible light weighted feature maps; performing dot product operations on multiple infrared region probability maps with the first infrared feature map respectively to obtain multiple infrared weighted feature maps; performing addition operations on each visible light weighted feature map with the corresponding infrared weighted feature map to obtain multiple fused feature maps; and merging the multiple fused feature maps to obtain the final fused feature map. The process of merging multiple fusion feature maps involves: for key information regions, averaging the fusion features of multiple fusion feature maps to obtain the final fusion result; for non-key information regions, using the fusion features of the fusion feature map corresponding to the main convolution module as the final fusion result.

5. The dual-light fusion method based on a multi-scale region attention mechanism as described in claim 4, characterized in that: When performing thresholding operations on the visible light probability map and infrared probability map corresponding to the main convolution module, all probabilities greater than the threshold t are retained, and probabilities less than the threshold t are set to t. When performing thresholding operations on the visible light probability map and infrared probability map corresponding to the auxiliary convolution module, only all probabilities greater than the threshold t are retained, and probabilities less than the threshold t are set to 0. The process of merging multiple fusion feature maps includes: for the portion of the fusion feature map corresponding to the main convolution module that is above the threshold t, the average value of the fusion features of multiple fusion feature maps is calculated as the fusion result of the final fusion feature map; for the portion of the fusion feature map corresponding to the main convolution module that is below the threshold t, the fusion features of the fusion feature map corresponding to the main convolution module are used as the fusion result of the final fusion feature map.

6. The dual-light fusion method based on a multi-scale region attention mechanism as described in claim 4, characterized in that: The last convolutional module in the classification network is used as the main convolutional module, and the remaining one or more convolutional modules are used as auxiliary convolutional modules.

7. The dual-light fusion method based on a multi-scale region attention mechanism as described in claim 4, characterized in that: The second visible light feature map and the second infrared feature map are processed, specifically including: upsampling the second visible light feature map and the second infrared feature map to the same size as the visible light image and the infrared image registered with it, respectively, and performing exponential normalization to obtain the visible light probability map and the infrared probability map.

8. A dual-light fusion device based on a multi-scale region attention mechanism, characterized in that, include: The encoding network module receives a visible light image and an infrared image registered with it, extracts features from the visible light image and the infrared image respectively, and outputs a first visible light feature map and a first infrared feature map to the fusion module. The encoding network module includes multiple encoding modules connected in sequence. Dense connections are introduced into the encoding network, specifically including connecting the output of each encoding module with a pooling layer in the encoding network to the input of all subsequent encoding modules. The classification network module is used to receive a visible light image and an infrared image registered with it, extract features from the infrared image and the visible light image respectively, and output a second visible light feature map and a second infrared feature map to the fusion module. The fusion module is used to receive the first visible light feature map and the first infrared feature map output by the encoding network module and the second visible light feature map and the second infrared feature map output by the classification network module, and perform a fusion operation to obtain a fused feature map and output it to the decoding network module. A decoding network module is used to reconstruct the fused feature map to obtain a fused image. The decoding network module includes multiple decoding modules connected in sequence. Cross-layer connections are added between the encoding network and the decoding network. Specifically, the output of each encoding module with a pooling layer in the encoding network is connected to the input of the corresponding decoding module in the decoding network.

9. A dual-light fusion device based on a multi-scale region attention mechanism, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the dual-light fusion method as described in any one of claims 1 to 7.

Images