Image fusion method and system

Abstract

The invention discloses an image fusion method and system, relates to the technical field of image processing, and aims to at least solve the problems that key information is not recognized in a feature extraction process and two image detail features are difficult to fuse in an image fusion process, so that a generated fusion image is blurred, and the key information of the image is easy to lose. The method comprises the following steps: acquiring a visible light image and an infrared image of a target object; performing feature extraction on the first target feature of the visible light image and the second target feature of the infrared image by adopting a feature extraction module to correspondingly obtain a first feature extraction image and a second feature extraction image; the feature extraction module comprises a first unit, a second unit and a third unit; and performing feature reconstruction on the first target feature in the first feature extraction image and the second target feature in the second feature extraction image by adopting a feature reconstruction module to obtain a feature fusion image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image fusion method and system. Background Technology

[0002] Advances in optical imaging technology have provided more reliable image information sources for target detection and recognition. However, maintaining stable image fusion effects under varying weather, lighting, and imaging conditions is challenging, thus impacting target detection and recognition. Visible light images offer high color contrast and richer texture features, but they are highly sensitive to lighting conditions; insufficient or excessive light intensity can affect image quality and visible details. Infrared sensors capture the thermal radiation information of targets. Compared to visible light, they have stronger penetrating power, allowing for the acquisition of target contour images even in harsh environments such as fog, nighttime, or low-light conditions. However, their lower image resolution and fewer visible gradient texture features result in blurred or missing key features in the fused image. Summary of the Invention

[0003] This invention provides an image fusion method and system to at least solve the problems of failing to identify key information during feature extraction and the difficulty in fusing two types of image detail features during image fusion, resulting in blurry fused images that easily lose key image information. The technical solution of this invention is as follows:

[0004] According to a first aspect of the present invention, an image fusion method is provided, comprising acquiring a visible light image and an infrared image of a target object; employing a feature extraction module to extract features from a first target feature in the visible light image and a second target feature in the infrared image, respectively, to obtain a first feature extraction image and a second feature extraction image; the feature extraction module includes a first unit, a second unit, and a third unit; the first unit is used to extract multi-scale features; the second unit is used to extract nonlinear features; the third unit is used to perform feature extraction according to an attention mechanism; and employing a feature reconstruction module to reconstruct features from the first target feature in the first feature extraction image and the second target feature in the second feature extraction image, to obtain a feature fusion image.

[0005] In one implementation, a feature extraction module is used to extract features from a first target feature in a visible light image and a second target feature in an infrared image, respectively, to obtain a first feature extraction image and a second feature extraction image. This includes: calling a first unit to extract multi-scale features from the visible light image and the infrared image, respectively, to obtain a first multi-scale feature image of the visible light image and a second multi-scale feature image corresponding to the infrared image; calling a second unit to extract nonlinear features from the first multi-scale feature image and the second multi-scale feature image, respectively, to obtain a first nonlinear feature image corresponding to the first multi-scale feature image and a second nonlinear feature image corresponding to the second multi-scale feature image; and calling a third unit to use an attention mechanism to extract the first target feature from the first nonlinear feature image and the second target feature from the second nonlinear feature image, respectively, to obtain the first feature extraction image and the second feature extraction image.

[0006] In this implementation, the first unit is invoked to extract image features at different scales from visible light and infrared images, thereby realizing the perception and fusion of contextual information at different scales of the images; the second unit is invoked to perform nonlinear transformations on the feature information in the obtained first and second multi-scale feature images, respectively, to enhance the expressive power of the features and obtain deep feature extraction results; the third unit is invoked to apply an attention mechanism to the first and second nonlinear feature images after nonlinear transformation to enhance the representational power of image feature information and highlight key feature information.

[0007] In another implementation, the first unit includes two first convolutional kernels and three convolutional kernels with different dilation rates; the first convolutional kernel is the first convolutional scale; the first unit is called to extract multi-scale features of visible light images and infrared images respectively, to obtain a first multi-scale feature image of the visible light image and a second multi-scale feature image corresponding to the infrared image, including: connecting one convolutional kernel and three convolutional kernels with different dilation rates in parallel to form a first sub-unit; calling the first sub-unit to extract feature information of multiple different scales of visible light images and infrared images respectively, to obtain multiple first sub-feature images and second sub-feature images containing feature information of different scales respectively; calling another first convolutional kernel to concatenate the features in the first sub-feature image and to concatenate the features in the second sub-feature image respectively, to obtain the first multi-scale feature image and the second multi-scale feature image respectively.

[0008] In this implementation, the first unit uses convolution kernels with different dilation rates to capture local details and global context simultaneously, enhancing the perception and fusion of contextual information at different scales; another first convolution kernel is invoked for information fusion and dimensionality compression, effectively enhancing the sensitivity to image details, edges and target regions, and improving the feature extraction effect of the image in complex scenes.

[0009] In another implementation, the second unit includes a first residual block, a second residual block, and a third residual block; each of the first, second, and third residual blocks includes a second convolutional kernel, a third convolutional kernel, two first convolutional kernels, and a first activation function; the second convolutional kernel includes a first activation function and a first convolutional scale; the third convolutional kernel includes a first activation function and a second convolutional scale; calling the second unit to extract nonlinear features from the first multi-scale feature image and the second multi-scale feature image respectively, to obtain the first nonlinear feature image corresponding to the first multi-scale feature image and the second nonlinear feature image corresponding to the second multi-scale feature image, includes: calling the first residual block to perform a first feature nonlinear transformation on the first multi-scale feature image to obtain a first visible light transformation function; adding the first eigenvalue mapped from the first multi-scale feature image to the first visible light transformation function to obtain a first visible light residual function; and performing a first feature nonlinear transformation on the second multi-scale feature image to obtain a first infrared transformation function; adding the second eigenvalue mapped from the second multi-scale feature image to the first infrared transformation function. The transformation functions are added together to obtain a first infrared residual function; the second residual block is called to perform a second characteristic nonlinear transformation on the first visible light residual function to obtain a second visible light transformation function; the first visible light residual function and the second visible light transformation function are added together to obtain a second visible light residual function; the first infrared residual function is then subjected to a second characteristic nonlinear transformation to obtain a second infrared transformation function; the first infrared residual function and the second infrared transformation function are added together to obtain a second infrared residual function; the third residual block is called to perform a third characteristic nonlinear transformation on the second visible light residual function to obtain a third visible light transformation function; the second visible light residual function and the third visible light transformation function are added together to obtain a third visible light residual function, and a first nonlinear feature image corresponding to the third visible light residual function is obtained; the second infrared residual function is then subjected to a third characteristic nonlinear transformation to obtain a third infrared transformation function; the second infrared residual function and the third infrared transformation function are added together to obtain a third infrared residual function, and a second nonlinear feature image corresponding to the third infrared residual function is obtained.

[0010] In this embodiment, the second unit uses three residual blocks to extract deep information of image features. Each residual block adopts a skip connection structure, which directly adds the input image features to the nonlinearly transformed image features, effectively preserving the input feature information while further enhancing the expressive power of the feature. Therefore, using residual blocks for feature extraction can extract the deep semantic information of the image while ensuring that the structure and information of the image are maintained during feature transfer.

[0011] In another implementation, the third unit includes a feature weight determination unit and a spatial weight determination unit. Calling the third unit and employing an attention mechanism to extract the first target feature of the first nonlinear feature image and the second target feature of the second nonlinear feature image, respectively, to obtain a first feature extraction image and a second feature extraction image, includes: calling the feature weight determination unit to identify features in the first and second nonlinear feature images using a first attention mechanism to determine the first feature weight of each feature in the first nonlinear feature image and the second feature weight of each feature in the second nonlinear feature image; calling the spatial weight determination unit to determine the first spatial weight of each feature of the first feature weight in the image region of the first nonlinear feature image using a second attention mechanism, and to determine the second spatial weight of each feature of the second feature weight in the image region of the second nonlinear feature image using a second attention mechanism; extracting the first target feature of the first nonlinear feature image based on the first spatial weight and the first feature weight, to obtain a first feature extraction image; and extracting the second target feature of the second nonlinear feature image based on the second spatial weight and the second feature weight, to obtain a second feature extraction image.

[0012] In this embodiment, the third unit uses a feature weight determination unit and a spatial weight determination unit to perform multiple weighting on the image features, thereby determining more critical image features and more critical spatial regions.

[0013] In another implementation, the feature weight determination unit includes: a first average pooling unit, a first max pooling unit, a second activation function, and a first convolution scale; the feature weight determination unit is invoked to identify features in the first nonlinear feature image and the second nonlinear feature image using a first attention mechanism, respectively, to determine the first feature weight of each feature in the first nonlinear feature image and the second feature weight of each feature in the second nonlinear feature image, including: invoking the first average pooling unit to determine the first global context information of the first nonlinear feature image; and invoking the first max pooling unit to determine the first salient local features of the first nonlinear feature image; and then... The first feature image is obtained by concatenating the first feature information with the first salient local features; the second feature weight information is obtained by calling the second activation function and the first convolution scale, and weighting the first nonlinear feature image according to the first feature weight information; the second global context information of the second nonlinear feature image is obtained by calling the first average pooling unit; the second salient local features of the second nonlinear feature image are obtained by calling the first max pooling unit; the second global context information is concatenated with the second salient local features to obtain the second feature weight information; the second feature weight information of the second nonlinear feature image is obtained by calling the second activation function and the first convolution scale, and weighting the second nonlinear feature image according to the second feature weight information.

[0014] In this embodiment, by utilizing the feature attention mechanism of the feature weight determination unit, key features with high weights in the image are determined, and redundant features related to background or noise with low weights are suppressed. In this way, the second nonlinear feature image is weighted according to the first feature weight information to focus on the key features in the image.

[0015] In another implementation, the spatial weight determination unit includes: a second average pooling unit, a second max pooling unit, a second activation function, and a third convolution scale. The spatial weight determination unit is invoked to determine the first spatial weight of each feature of the first feature weight in the image region of the first nonlinear feature image using a second attention mechanism. The second spatial weight of each feature of the second feature weight in the image region of the second nonlinear feature image is determined using the second attention mechanism, including: invoking the second average pooling unit to determine the first spatial average value of each feature of the first feature weight; and invoking the second max pooling unit to determine the first spatial maximum value of each feature of the first feature weight; and concatenating the first spatial average value and the first spatial maximum value. The process involves obtaining first spatial weight information; calling a second activation function and a third convolution scale, and then weighting each feature of the first feature weight in the image region of the first nonlinear feature image based on the first spatial weight information to obtain the first spatial weight; calling a second average pooling unit to determine the second spatial average value of each feature of the second feature weight; calling a second max pooling unit to determine the second spatial maximum value of each feature of the second feature weight; concatenating the second spatial average value and the second spatial maximum value to obtain second spatial weight information; and then calling a second activation function and a third convolution scale, and then weighting each feature of the second feature weight in the image region of the second nonlinear feature image based on the second spatial weight information to obtain the second spatial weight.

[0016] In this embodiment, by utilizing the spatial attention mechanism of the spatial weight determination unit, the high-weight key attention positions in the feature image are determined, and irrelevant backgrounds are suppressed; this complements the first feature weight after feature weighting, thereby enhancing the perception of image structure.

[0017] In another implementation, the feature reconstruction module includes a first reconstruction residual block, a second reconstruction residual block, a third reconstruction residual block, a fourth reconstruction residual block, a fifth reconstruction residual block, and a third activation function; the first, second, third, fourth, and fifth reconstruction residual blocks have the same structure as the first, second, and third residual blocks; using the feature reconstruction module, feature reconstruction is performed on the first target feature in the first feature extraction image and the second target feature in the second feature extraction image to obtain a feature fusion image, including: calling the first reconstruction residual block to perform image fusion on the first and second feature extraction images to obtain a first fused feature image; performing a first reconstruction nonlinear transformation on the first and second target features to obtain first reconstruction residual features; adding each feature in the first fused feature image to the first reconstruction residual features to obtain a first fused image including the first reconstruction features; calling the second reconstruction residual block... A second reconstruction nonlinear transformation is applied to the first reconstructed feature to obtain a second reconstructed residual feature. The first reconstructed feature and the second reconstructed residual feature are added together to obtain a second fused image including the second reconstructed feature. A third reconstruction residual block is called to apply a third reconstruction nonlinear transformation to the second reconstructed feature to obtain a third reconstructed residual feature. The second reconstructed feature and the third reconstructed residual feature are added together to obtain a third fused image including the third reconstructed feature. A fourth reconstruction residual block is called to apply a fourth reconstruction nonlinear transformation to the third reconstructed feature to obtain a fourth reconstructed residual feature. The third reconstructed feature and the fourth reconstructed residual feature are added together to obtain a fourth fused image including the fourth reconstructed feature. A fifth reconstruction residual block is called to apply a fifth reconstruction nonlinear transformation to the fourth reconstructed feature to obtain a fifth reconstructed residual feature. The fourth reconstructed feature and the fifth reconstructed residual feature are added together to obtain a fifth fused image including the fifth reconstructed feature. A third activation function is called to adjust the scale range of the fifth fused image to obtain a feature fused image.

[0018] In this embodiment, the reconstruction residual block in the feature reconstruction module has the same structure as the residual block in the feature extraction module, ensuring that the visible light feature image to be fused and the infrared feature image are spatially aligned; the image features are gradually fused by five layers of feature reconstruction residual blocks, thereby improving the quality and stability of the fused image.

[0019] In another implementation, multiple sets of images are acquired and preprocessed. Based on the preprocessed image data, an image MSRS dataset is constructed. Each image set represents a visible light image and an infrared image of the same target object. An image fusion network model is constructed based on a feature extraction module and a feature reconstruction module. The image fusion network model is trained based on the image MSRS dataset until the loss function converges, completing the model training. A set of visible light images and infrared images are input into the image fusion network model, and the target feature fusion image is output.

[0020] According to a second aspect of the present invention, an image fusion system is provided, the image fusion system comprising: a feature extraction module and a feature reconstruction module, the image fusion system being obtained using the image fusion method as described in any of the implementations of the first aspect above.

[0021] According to a third aspect of the present invention, an image fusion apparatus is provided, the apparatus comprising:

[0022] The image acquisition unit is configured to acquire visible light and infrared images of the target object.

[0023] The feature extraction unit is configured to use the feature extraction module to extract features from the first target features of the visible light image and the second target features of the infrared image, respectively, to obtain the first feature extraction image and the second feature extraction image. The feature extraction module includes a first unit, a second unit and a third unit. The first unit is used to extract multi-scale features. The second unit is used to extract nonlinear features. The third unit is used to perform feature extraction according to the attention mechanism.

[0024] The image fusion unit is configured to use a feature reconstruction module to reconstruct the features of the first target in the first feature extraction image and the second target in the second feature extraction image to obtain a feature fusion image.

[0025] According to a fourth aspect of the present invention, an image fusion apparatus is provided, the apparatus being configured to perform an image fusion method as described in the first aspect and any possible implementation thereof.

[0026] According to a fifth aspect of the present invention, a computer-readable storage medium is provided, on which instructions are stored, such that when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform an image fusion method as described in the first aspect and any possible implementation thereof.

[0027] According to a sixth aspect of the present disclosure, a computer program product is provided, the computer program product including computer instructions that, when executed on an electronic device, cause the electronic device to perform the image fusion method of the first aspect and any possible implementation thereof.

[0028] The technical solution provided by the embodiments of the present invention brings at least the following beneficial effects: A feature extraction module is used to extract features at multiple different scales from visible light and infrared images respectively. To fully identify image features in various regions of the image while ensuring image clarity, a first unit is used to extract multi-scale features of the image. Furthermore, to preserve the image's results and details during feature transfer while extracting deep semantic information, a second unit is used to extract nonlinear features of the image. To more accurately identify key image features and key image regions, a third unit is used to extract features according to an attention mechanism.

[0029] Based on the above image fusion method, different image feature extraction methods are used to extract key image features and image detail features of different dimensions of the image multiple times. Based on the extracted features, the first feature extraction image and the second feature extraction image are fused to obtain a feature fusion image with prominent target object and clear image.

[0030] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0031] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application, and do not constitute an undue limitation of this application.

[0032] Figure 1 This is a schematic diagram of an image fusion system according to an exemplary embodiment. Figure 1 ;

[0033] Figure 2 This is a schematic diagram of an image fusion system according to an exemplary embodiment. Figure 2 ;

[0034] Figure 3 This is a flowchart of an image fusion method according to an exemplary embodiment. Figure 1 ;

[0035] Figure 4 This is a flowchart of an image fusion method according to an exemplary embodiment. Figure 2 ;

[0036] Figure 5 This is a schematic diagram illustrating a multi-scale feature extraction structure for a first unit according to an exemplary embodiment;

[0037] Figure 6 This is a schematic diagram illustrating a residual block structure of a second unit according to an exemplary embodiment;

[0038] Figure 7 This is a schematic diagram illustrating an attention mechanism feature extraction structure for a third unit according to an exemplary embodiment;

[0039] Figure 8 This is a schematic diagram of a visible light and infrared image fusion network according to an exemplary embodiment;

[0040] Figure 9 This is a chart illustrating an objective evaluation index for a nighttime scene, according to an exemplary embodiment.

[0041] Figure 10 This is a chart illustrating an objective evaluation index for a daytime scene, according to an exemplary embodiment.

[0042] Figure 11 This is a schematic diagram illustrating a nighttime scene blending effect according to an exemplary embodiment;

[0043] Figure 12 This is a schematic diagram illustrating a daytime scene blending effect according to an exemplary embodiment;

[0044] Figure 13 This is a schematic diagram illustrating an image fusion apparatus according to an exemplary embodiment;

[0045] Figure 14 This is a schematic diagram of an image fusion device according to an exemplary embodiment. Detailed Implementation

[0046] To enable those skilled in the art to better understand the technical solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0047] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0048] Before providing a detailed description of the image fusion method provided in the embodiments of this application, we will first briefly introduce the application scenarios involved in the embodiments of this application.

[0049] Advances in optical imaging technology have provided more reliable image information sources for target detection and recognition. However, maintaining stable image fusion effects under varying weather, lighting, and imaging conditions is challenging, thus impacting target detection and recognition. Visible light images offer high color contrast and richer texture features, but they are highly sensitive to lighting conditions; insufficient or excessive light intensity can affect image quality and visible details. Infrared sensors capture the thermal radiation information of targets. Compared to visible light, they have stronger penetrating power, allowing for the acquisition of target contour images even in harsh environments such as fog, nighttime, or low-light conditions. However, their lower image resolution and fewer visible gradient texture features result in blurred or missing key features in the fused image.

[0050] Research has revealed two main problems in image feature extraction and fusion: First, during feature extraction, the thermal radiation information of infrared images and the detailed information of visible light images are not fully preserved, resulting in the loss of key information in the fused image and affecting the accuracy of subsequent target detection and recognition. Second, during image fusion, it is difficult to balance the structural information of infrared images and the texture details of visible light images, leading to blurring or loss of detail in the fused image.

[0051] To address the aforementioned issues, this application provides an image fusion method. A feature extraction module extracts features at multiple scales from both visible light and infrared images. To fully identify image features in various regions while maintaining image clarity, a first unit extracts multi-scale features. Furthermore, to preserve image details and results during feature transfer while extracting deep semantic information, a second unit extracts nonlinear features. Finally, to more accurately identify key image features and regions, a third unit performs feature extraction using an attention mechanism.

[0052] Based on the above image fusion method, different image feature extraction methods are used to extract key image features and image detail features of different dimensions of the image multiple times. Based on the extracted features, the first feature extraction image and the second feature extraction image are fused to obtain a feature fusion image with prominent target object and clear image.

[0053] Secondly, the implementation architecture involved in this application will be briefly introduced below.

[0054] Figure 1 This is a schematic diagram of an image fusion system provided in this application. Figure 1 As shown, the image fusion system includes a feature extraction module 11 and a feature reconstruction module 12.

[0055] The feature extraction module 11 and the feature reconstruction module 12 are connected via communication.

[0056] The feature extraction module 11 is configured to extract features from the first target feature of the visible light image and the second target feature of the infrared image, respectively, to obtain the first feature extraction image and the second feature extraction image.

[0057] like Figure 2 As shown, the feature extraction module 11 includes a first unit 111, a second unit 112, and a third unit 113.

[0058] The first unit 111 is configured to extract multi-scale features.

[0059] The second unit 112 is configured to extract nonlinear features.

[0060] The third unit 113 is configured to perform feature extraction according to the attention mechanism.

[0061] The feature reconstruction module 12 is configured to reconstruct the features of the first target in the first feature extraction image and the second target in the second feature extraction image to obtain a feature fusion image.

[0062] For ease of understanding, the image fusion method provided in this application will be described in detail below with reference to the accompanying drawings.

[0063] Figure 3 This is a flowchart illustrating an image fusion method according to an exemplary embodiment, such as... Figure 3 As shown, the image fusion method includes the following steps.

[0064] S31, acquire the visible light image and infrared image of the target object.

[0065] Visible light images are acquired using an optical imaging sensor, and infrared images are acquired using an infrared imaging sensor.

[0066] A set of visible light and infrared images of the target are acquired, and their time, location, and lighting conditions are the same. The visible light and infrared images of each other are matched.

[0067] S32, a feature extraction module is used to extract features from the first target feature of the visible light image and the second target feature of the infrared image, respectively, to obtain the first feature extraction image and the second feature extraction image.

[0068] The feature extraction module consists of a first unit, a second unit, and a third unit.

[0069] The first unit is used to extract multi-scale features, the second unit is used to extract nonlinear features, and the third unit is used to extract features according to the attention mechanism.

[0070] The first unit is invoked to extract image features at different scales from visible light and infrared images, enabling the perception and fusion of contextual information at different scales. The second unit is invoked to perform nonlinear transformations on the feature information in the first and second multi-scale feature images, respectively, to enhance the expressive power of the features and obtain deep feature extraction results. The third unit is invoked to apply an attention mechanism to the first and second nonlinear feature images after nonlinear transformation to enhance the representation of image feature information and highlight key feature information.

[0071] Specifically, such as Figure 4 As shown, steps S321 to S323 are specifically implemented for step S32.

[0072] S321, the first unit is called to extract multi-scale features of the visible light image and the infrared image respectively, to obtain the first multi-scale feature image of the visible light image and the second multi-scale feature image corresponding to the infrared image.

[0073] The first unit consists of two first convolution kernels and three convolution kernels with different dilatations. The first convolution kernel is the first convolution scale.

[0074] In one implementation, a first sub-unit is formed by connecting one convolutional kernel and three convolutional kernels with different dilation rates in parallel. The first sub-unit is then used to extract feature information at multiple different scales from the visible light image and the infrared image, respectively, resulting in multiple first sub-feature images and second sub-feature images containing feature information at different scales. Another first convolutional kernel is then used to concatenate the features in the first sub-feature images and the features in the second sub-feature images, respectively, to obtain the first multi-scale feature image and the second multi-scale feature image.

[0075] Specifically, the multi-scale feature extraction structure of the first unit is as follows: Figure 5 As shown, the first convolutional kernel is a 1x1 convolutional kernel. The three convolutional kernels with different dilatations are 3x3 convolutional kernels with dilatations of 2, 4, and 6, respectively. A first sub-unit is formed by connecting a 1x1 convolutional kernel and three 3x3 convolutional kernels with dilatations of 2, 4, and 6 in parallel. This sub-unit extracts feature information at multiple scales from the visible light and infrared images, resulting in multiple first and second sub-feature images containing feature information at different scales. Finally, another 1x1 convolutional kernel is used to concatenate the features in the first sub-feature image and the features in the second sub-feature image, respectively, to obtain the first multi-scale feature image and the second multi-scale feature image.

[0076] S322, the second unit is called to extract the nonlinear features of the first multi-scale feature image and the second multi-scale feature image respectively, to obtain the first nonlinear feature image corresponding to the first multi-scale feature image and the second nonlinear feature image corresponding to the second multi-scale feature image.

[0077] The first nonlinear feature image represents the image containing the first nonlinear feature extraction result after undergoing the first nonlinear transformation.

[0078] The second nonlinear feature image represents the image containing the results of the second nonlinear feature extraction after undergoing the second nonlinear transformation.

[0079] The second unit includes a first residual block, a second residual block, and a third residual block.

[0080] The first residual block, the second residual block, and the third residual block each include a second convolution kernel, a third convolution kernel, two first convolution kernels, and a first activation function.

[0081] The second convolutional kernel includes a first activation function and a first convolutional scale. The third convolutional kernel includes a first activation function and a second convolutional scale.

[0082] In one implementation, a first residual block is invoked to perform a first feature nonlinear transformation on a first multi-scale feature image to obtain a first visible light transformation function; the first eigenvalue mapped from the first multi-scale feature image is added to the first visible light transformation function to obtain a first visible light residual function; and the first feature nonlinear transformation is performed on a second multi-scale feature image to obtain a first infrared transformation function; the second eigenvalue mapped from the second multi-scale feature image is added to the first infrared transformation function to obtain a first infrared residual function.

[0083] The second residual block is invoked to perform a second characteristic nonlinear transformation on the first visible light residual function to obtain a second visible light transformation function; the first visible light residual function and the second visible light transformation function are added together to obtain a second visible light residual function; and the first infrared residual function is performed a second characteristic nonlinear transformation on the first infrared residual function to obtain a second infrared transformation function; the first infrared residual function and the second infrared transformation function are added together to obtain a second infrared residual function.

[0084] The third residual block is invoked to perform a third characteristic nonlinear transformation on the second visible light residual function to obtain a third visible light transformation function; the second visible light residual function is added to the third visible light transformation function to obtain a third visible light residual function, and a first nonlinear feature image corresponding to the third visible light residual function is obtained; and the third characteristic nonlinear transformation is performed on the second infrared residual function to obtain a third infrared transformation function; the second infrared residual function is added to the third infrared transformation function to obtain a third infrared residual function, and a second nonlinear feature image corresponding to the third infrared residual function is obtained.

[0085] Specifically, the residual block structure of the second unit is as follows: Figure 6 As shown, the first residual block, the second residual block, and the third residual block each consist of a second convolutional kernel consisting of a 1×1 convolutional kernel and a ReLU activation function, a third convolutional kernel consisting of a 3×3 convolutional kernel and a ReLU activation function, two 1×1 convolutional kernels, and a ReLU activation function.

[0086] Assuming the first or second multi-scale feature image of the first residual block is x, after performing a nonlinear transformation using a second convolution kernel, a third convolution kernel, and a first convolution kernel, the result is added to the result of a 1*1 convolution. Finally, the transformed feature f(x) is obtained through the ReLU activation function, and the output of the first residual block is x+f(x).

[0087] The input to the second residual block is x+f(x), which is transformed by a nonlinear transformation with the same structure as the first residual block to obtain f(x+f(x)). The output of the second residual block is x+f(x)+f(x+f(x)).

[0088] The input to the third residual block is x+f(x)+f(x+f(x)), which, after undergoing a nonlinear transformation with the same structure as the first residual block, yields f(x+f(x)+f(x+f(x))). The output of the third residual block is x+f(x)+f(x+f(x))+f(x+f(x)+f(x+f(x))).

[0089] Understandably, residual blocks are used to alleviate gradient vanishing and performance degradation during deep network training. By introducing skip connections, the input features are directly added to the nonlinearly transformed features, effectively preserving the original feature information and enhancing feature representation capabilities. Specifically, 1x1 convolutions retain key information while compressing and adjusting the number of channels, 3x3 convolutional modules are used for spatial local feature extraction, such as texture and edge information, and the ReLU activation function is responsible for enhancing nonlinear representation, allowing the model to learn more complex nonlinear mappings. In image fusion tasks, residual blocks not only help extract deep semantic information from images but also maintain the image structure and details during feature transfer, improving the quality and stability of the fused image.

[0090] S323, call the third unit, and use the attention mechanism to extract the first target feature of the first nonlinear feature image and the second target feature of the second nonlinear feature image respectively, to obtain the first feature extraction image and the second feature extraction image.

[0091] The third unit includes the feature weight determination unit and the spatial weight determination unit.

[0092] In one implementation, the attention mechanism feature extraction structure of the third unit is as follows: Figure 7 As shown.

[0093] The weight determination unit includes: a first average pooling unit, a first max pooling unit, a second activation function, and a first convolution scale.

[0094] The spatial weight determination unit includes: a second average pooling unit, a second max pooling unit, a second activation function, and a third convolution scale.

[0095] Specifically, the weight determination unit includes: a first average pooling unit, a first max pooling unit, a sigmoid activation function, and a 1×1 convolutional kernel. The spatial weight determination unit includes: a second average pooling unit, a second max pooling unit, a sigmoid activation function, and a 7×7 convolutional kernel.

[0096] First, the feature weight determination unit is invoked, and the first attention mechanism is used to identify the features in the first nonlinear feature image and the second nonlinear feature image respectively, so as to determine the first feature weight of each feature in the first nonlinear feature image and the second feature weight of each feature in the second nonlinear feature image.

[0097] In a specific implementation, the first average pooling unit is invoked to determine the first global context information of the first nonlinear feature image; and the first max pooling unit is invoked to determine the first salient local feature of the first nonlinear feature image; the first global context information and the first salient local feature are concatenated to obtain the first feature weight information; and the second activation function and the first convolution scale are invoked to weight the first nonlinear feature image according to the first feature weight information to obtain the first feature weight.

[0098] In addition, the first average pooling unit is called to determine the second global context information of the second nonlinear feature image; and the first max pooling unit is called to determine the second salient local features of the second nonlinear feature image; the second global context information and the second salient local features are concatenated to obtain the second feature weight information; the second activation function and the first convolution scale are called to weight the second nonlinear feature image according to the second feature weight information to obtain the second feature weight.

[0099] In this specific implementation, a first average pooling unit captures the global contextual information of the feature image, a first max pooling unit highlights the significant local features of the feature image, and the results of the double pooling are concatenated or added. Then, feature weights are generated using a 1×1 convolution and a Sigmoid activation function, where the Sigmoid activation function compresses the weights. The nonlinear feature image input to the third unit is then weighted according to the generated feature weights.

[0100] Secondly, the spatial weight determination unit is invoked, and the first spatial weight of each feature of the first feature weight in the image region of the first nonlinear feature image is determined by the second attention mechanism, and the second spatial weight of each feature of the second feature weight in the image region of the second nonlinear feature image is determined by the second attention mechanism.

[0101] In a specific implementation, the second average pooling unit is invoked to determine the first spatial average value of each feature of the first feature weight; and the second max pooling unit is invoked to determine the first spatial maximum value of each feature of the first feature weight; the first spatial average value and the first spatial maximum value are concatenated to obtain the first spatial weight information; the second activation function and the third convolution scale are invoked to re-weight each feature of the first feature weight in the image region located in the first nonlinear feature image according to the first spatial weight information to obtain the first spatial weight.

[0102] Then, the second average pooling unit is invoked to determine the second spatial average value of each feature of the second feature weight. The second max pooling unit is then invoked to determine the second spatial maximum value of each feature of the second feature weight. The second spatial average value and the second spatial maximum value are concatenated to obtain the second spatial weight information. The second activation function and the third convolution scale are then invoked, and based on the second spatial weight information, each feature of the second feature weight is weighted again within its corresponding image region in the second nonlinear feature image to obtain the second spatial weight.

[0103] In this specific implementation, the feature image after weighting the feature weights is spatially weighted again along the channel axis. A second average pooling unit is used to determine the mean of all channels at each spatial location, thereby identifying globally salient regions. A second max pooling unit is used to extract the maximum value of all channels at each location, thereby capturing the strongest local activation at that location and highlighting details or outliers. After concatenating or adding the double pooling results, spatial weights are generated by a 7×7 convolution kernel + sigmoid activation function. The 7×7 convolution kernel is used to capture wide-area spatial relationships, fusing contextual information from surrounding pixels and avoiding over-focusing on isolated points. Finally, the nonlinear feature image input to the third unit is weighted again based on the generated spatial weights, building upon the first layer of feature weight weighting.

[0104] Third, based on the first spatial weight and the first feature weight, the first target feature of the first nonlinear feature image is extracted to obtain the first feature extraction image; and based on the second spatial weight and the second feature weight, the second target feature of the second nonlinear feature image is extracted to obtain the second feature extraction image.

[0105] Understandably, after feature weighting, the region of the target object, such as a "pedestrian," is enhanced in the infrared feature image, while the texture is preserved in the visible light feature image. In the spatial weight determination unit, the average pooling response activates the entire pedestrian region on a large scale, while the max pooling response highly activates local areas such as the pedestrian boundary and collar texture. The 7×7 convolution combines the two, and the final spatial weight matrix highlights the pedestrian outline and texture details. Through a dual attention mechanism of features and space, "which features are useful" is first selected, and then "where they are useful" is located, thereby achieving progressive optimization of the feature map from global to local.

[0106] S33, using a feature reconstruction module, the first target feature in the first feature extraction image and the second target feature in the second feature extraction image are reconstructed to obtain a feature fusion image.

[0107] The feature reconstruction module includes a first reconstruction residual block, a second reconstruction residual block, a third reconstruction residual block, a fourth reconstruction residual block, a fifth reconstruction residual block, and a third activation function; the first reconstruction residual block, the second reconstruction residual block, the third reconstruction residual block, the fourth reconstruction residual block, and the fifth reconstruction residual block have the same structure as the first residual block, the second residual block, and the third residual block.

[0108] Ensure that the structure of the reconstructed residual block is the same as that used during feature extraction, so as to ensure spatial alignment between the infrared feature image and the visible light feature image to be fused.

[0109] The third activation function is the tanh activation function, which is used to adjust the range of the fused image and prevent pixel values ​​from exceeding the visualization range.

[0110] In one implementation, a first reconstruction residual block is invoked to fuse the first feature extraction image and the second feature extraction image to obtain a first fused feature image. A first reconstruction nonlinear transformation is then performed on the first target feature and the second target feature to obtain first reconstruction residual features. Each feature in the first fused feature image is added to the first reconstruction residual features to obtain a first fused image including the first reconstruction features. A second reconstruction residual block is invoked to perform a second reconstruction nonlinear transformation on the first reconstruction features to obtain second reconstruction residual features. The first reconstruction features are added to the second reconstruction residual features to obtain a second fused image including the second reconstruction features. A third reconstruction residual block is invoked to perform a third reconstruction nonlinear transformation on the second reconstruction features to obtain third reconstruction residual features. The second reconstruction features are added to the third reconstruction residual features to obtain a third fused image including the third reconstruction features. A fourth reconstruction residual block is invoked to perform a fourth reconstruction nonlinear transformation on the third reconstruction features to obtain fourth reconstruction residual features. The third reconstruction features are added to the fourth reconstruction residual features to obtain a fourth fused image including the fourth reconstruction features. The fifth reconstruction residual block is invoked to perform a fifth reconstruction nonlinear transformation on the fourth reconstruction features, resulting in the fifth reconstruction residual features. The fourth reconstruction features and the fifth reconstruction residual features are then added together to obtain the fifth fused image, which includes the fifth reconstruction features. The third activation function is then invoked to adjust the scale range of the fifth fused image, resulting in the feature fused image.

[0111] Understandably, by progressively restoring high-frequency details, edges, and textures through five concatenated reconstruction residual blocks, and correcting them step by step from global to local, the blurring problem of one-time reconstruction is avoided, and the final output is a fused image with rich details and clear textures.

[0112] Optionally, multiple sets of images are acquired and preprocessed. Based on the preprocessed image data, an image MSRS dataset is constructed. Each image set represents a visible light image and an infrared image of the same target object. An image fusion network model is constructed based on a feature extraction module and a feature reconstruction module. The image fusion network model is trained using the image MSRS dataset until the loss function converges, completing the model training. A set of visible light and infrared images is input into the image fusion network model, and the output is a target feature fused image.

[0113] In this embodiment, an image fusion network model is constructed based on the feature extraction module and feature reconstruction module of steps S21 to S23 described above.

[0114] First, prepare the dataset. After preprocessing pairs of infrared and visible light images, construct the MSRS dataset as an image fusion sample dataset.

[0115] Secondly, an image fusion network model is constructed. Based on the feature extraction module and the feature reconstruction module, an image fusion network model is constructed. The loss function of the model includes a pixel loss function and a gradient loss function.

[0116] Next, train the image fusion network model. Input the processed MSRS dataset into the image fusion network model and train the model until the loss function converges. At this point, the model training is considered complete, and the network model parameters are saved.

[0117] The pixel loss function guarantees pixel-level similarity between the fused image and the ideal result. The gradient loss function preserves image edge and structural information.

[0118] The loss function is shown in the following formula (1).

[0119]

[0120] in, For background pixel loss, For background region gradient loss, For significant pixel loss, For the gradient loss in the significant region, α is a proportional parameter that controls the loss balance.

[0121] Specifically, The formulas are shown in formulas (2) to (5) below.

[0122]

[0123] Where H and W are the height and width of the image, respectively, and ||·||1 represents the l1 norm. Represents the gradient operator, Indicates element-wise multiplication, I m To highlight the target mask, I f For the fused image, I ir For infrared images, I vi This is a visible light image.

[0124] Finally, the infrared and visible light images to be processed are input into the trained image fusion network model in pairs to obtain the fused image.

[0125] Based on the above implementation methods, an exemplary visible light and infrared image fusion network diagram is shown below. Figure 8 As shown.

[0126] Using the MSRS dataset as the source images, 400 pairs of images were selected for training and 40 pairs for validation. Four objective metrics were chosen to evaluate the fusion results: mutual information (MI), structural similarity-based identity (SSIM), peak signal-to-noise ratio (PSNR), and entropy (EN). MI measures the similarity between the fused image and the source images, i.e., how much information the fused image has acquired from the source images; the higher the mutual information, the better the fusion quality. A higher SSIM value indicates better consistency between the fused result and human visual features. PSNR measures the ratio of effective information to noise in the image to reflect whether the image is distorted; a higher value indicates better image fusion quality. EN is a metric for image sharpness and information content; a higher value indicates richer information extracted from the input image. The experimental results of fusion of infrared and visible light source images under two different scenarios are shown in Figures 9 and 10 below.

[0127] Specifically, the data from the U2Fusion fusion method and the image fusion method of this application are compared in nighttime scenes.

[0128] Comparing the MI (Mitigation Index) values, this application has a value of 1.7725, while U2Fusion has a value of 1.2243. Comparing the SSIM (Signal Score) values, this application has a value of 0.7531, while U2Fusion has a value of 0.7321. Comparing the PSNR (Power Response Rate) values, this application has a value of 22.5412, while U2Fusion has a value of 20.1321. Comparing the EN (Environmental Sensitivity Index) values, this application has a value of 5.3271, while U2Fusion has a value of 5.1253.

[0129] Compare the data from the U2Fusion fusion method and the image fusion method of this application in daytime scenes.

[0130] Comparing the MI (Mitigation Index) values, this application has a value of 2.1231, while U2Fusion has a value of 1.6243. Comparing the SSIM (Signal Score) values, this application has a value of 0.6232, while U2Fusion has a value of 0.5932. Comparing the PSNR (Power Response Rate) values, this application has a value of 15.6532, while U2Fusion has a value of 13.2334. Comparing the EN (Environmental Response Rate) values, this application has a value of 6.6518, while U2Fusion has a value of 6.4532.

[0131] The above four objective evaluation indicators show that the infrared and visible light image fusion effect of this application is better, and the image fusion effect is better after ensuring texture details and structural similarity.

[0132] A comparison of the fusion effects of the two algorithms in nighttime and daytime scenes, specifically as follows: Figure 11 and Figure 12 As shown, the image fusion method of this application produces a fused image with more refined details.

[0133] To achieve the above functions, the image fusion apparatus includes corresponding hardware structures and / or software modules for performing each function. Those skilled in the art will readily recognize that, based on the algorithmic steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is implemented in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0134] This disclosure also provides an embodiment such as Figure 13 The image fusion device shown includes an image acquisition unit 131, a feature extraction unit 132, and an image fusion unit 133.

[0135] The image acquisition unit 131 is configured to acquire visible light and infrared images of the target object.

[0136] The feature extraction unit 132 is configured to use the feature extraction module to extract features from the first target features of the visible light image and the second target features of the infrared image, respectively, to obtain the first feature extraction image and the second feature extraction image. The feature extraction module includes a first unit, a second unit and a third unit. The first unit is used to extract multi-scale features. The second unit is used to extract nonlinear features. The third unit is used to perform feature extraction according to the attention mechanism.

[0137] The image fusion unit 133 is configured to use a feature reconstruction module to reconstruct the first target feature in the first feature extraction image and the second target feature in the second feature extraction image to obtain a feature fusion image.

[0138] As one implementation, the feature extraction unit 132 is specifically configured to use a feature extraction module to extract features from the first target feature in the visible light image and the second target feature in the infrared image, respectively, to obtain a first feature extraction image and a second feature extraction image. This includes: calling a first unit to extract multi-scale features from the visible light image and the infrared image respectively, to obtain a first multi-scale feature image of the visible light image and a second multi-scale feature image corresponding to the infrared image; calling a second unit to extract nonlinear features from the first multi-scale feature image and the second multi-scale feature image respectively, to obtain a first nonlinear feature image corresponding to the first multi-scale feature image and a second nonlinear feature image corresponding to the second multi-scale feature image; and calling a third unit to use an attention mechanism to extract the first target feature of the first nonlinear feature image and the second target feature of the second nonlinear feature image respectively, to obtain the first feature extraction image and the second feature extraction image.

[0139] As one implementation, the first unit includes two first convolution kernels and three convolution kernels with different dilatation rates, and the first convolution kernel is the first convolution scale.

[0140] The feature extraction unit 132 is specifically configured to call the first unit to extract multi-scale features of the visible light image and the infrared image respectively, to obtain a first multi-scale feature image of the visible light image and a second multi-scale feature image corresponding to the infrared image, including: connecting a convolutional kernel and three convolutional kernels with different dilation rates in parallel to form a first sub-unit; calling the first sub-unit to extract feature information of multiple different scales of the visible light image and the infrared image respectively, to obtain multiple first sub-feature images and second sub-feature images containing feature information of different scales respectively; calling another first convolutional kernel to concatenate the features in the first sub-feature image and the features in the second sub-feature image respectively, to obtain the first multi-scale feature image and the second multi-scale feature image respectively.

[0141] In one implementation, the second unit includes a first residual block, a second residual block, and a third residual block. Each of the first, second, and third residual blocks includes a second convolutional kernel, a third convolutional kernel, two first convolutional kernels, and a first activation function. The second convolutional kernel includes a first activation function and a first convolutional scale. The third convolutional kernel includes a first activation function and a second convolutional scale.

[0142] The feature extraction unit 132 is specifically configured to call the second unit to extract nonlinear features from the first multi-scale feature image and the second multi-scale feature image respectively, to obtain the first nonlinear feature image corresponding to the first multi-scale feature image and the second nonlinear feature image corresponding to the second multi-scale feature image. This includes: calling the first residual block to perform a first feature nonlinear transformation on the first multi-scale feature image to obtain a first visible light transformation function; adding the first eigenvalue mapped from the first multi-scale feature image to the first visible light transformation function to obtain a first visible light residual function; performing a first feature nonlinear transformation on the second multi-scale feature image to obtain a first infrared transformation function; adding the second eigenvalue mapped from the second multi-scale feature image to the first infrared transformation function to obtain a first infrared residual function; and calling the second residual block to perform a second feature nonlinear transformation on the first visible light residual function to obtain a second visible light residual function. The process involves: 1) performing a function transformation; 2) adding the first visible light residual function to the second visible light transform function to obtain the second visible light residual function; 3) performing a second characteristic nonlinear transformation on the first infrared residual function to obtain the second infrared transform function; 4) adding the first infrared residual function to the second infrared transform function to obtain the second infrared residual function; 5) calling the third residual block to perform a third characteristic nonlinear transformation on the second visible light residual function to obtain the third visible light transform function; 6) adding the second visible light residual function to the third visible light transform function to obtain the third visible light residual function, and obtaining the first nonlinear feature image corresponding to the third visible light residual function; 7) performing a third characteristic nonlinear transformation on the second infrared residual function to obtain the third infrared transform function; 8) adding the second infrared residual function to the third infrared transform function to obtain the third infrared residual function, and obtaining the second nonlinear feature image corresponding to the third infrared residual function.

[0143] As one implementation method, the third unit includes a feature weight determination unit and a spatial weight determination unit.

[0144] The feature extraction unit 132 is specifically configured to: call the third unit to extract the first target feature of the first nonlinear feature image and the second target feature of the second nonlinear feature image using an attention mechanism, respectively, to obtain the first feature extraction image and the second feature extraction image, including: calling the feature weight determination unit to identify features in the first nonlinear feature image and the second nonlinear feature image using a first attention mechanism, to determine the first feature weight of each feature in the first nonlinear feature image and the second feature weight of each feature in the second nonlinear feature image; calling the spatial weight determination unit to determine the first spatial weight of each feature of the first feature weight in the image region of the first nonlinear feature image using a second attention mechanism, and to determine the second spatial weight of each feature of the second feature weight in the image region of the second nonlinear feature image using a second attention mechanism; extracting the first target feature of the first nonlinear feature image based on the first spatial weight and the first feature weight, to obtain the first feature extraction image; and extracting the second target feature of the second nonlinear feature image based on the second spatial weight and the second feature weight, to obtain the second feature extraction image.

[0145] As one implementation method, the feature weight determination unit includes: a first average pooling unit, a first max pooling unit, a second activation function, and a first convolution scale.

[0146] The feature extraction unit 132 is specifically configured to: call the feature weight determination unit, and use a first attention mechanism to identify features in the first nonlinear feature image and the second nonlinear feature image respectively, so as to determine the first feature weight of each feature in the first nonlinear feature image and the second feature weight of each feature in the second nonlinear feature image, including: calling the first average pooling unit to determine the first global context information of the first nonlinear feature image; and calling the first max pooling unit to determine the first salient local feature of the first nonlinear feature image; concatenating the first global context information with the first salient local feature to obtain the first feature weight information; calling the second activation function and the first convolution scale, and weighting the first nonlinear feature image according to the first feature weight information to obtain the first feature weight; and calling the first average pooling unit to determine the second global context information of the second nonlinear feature image; and calling the first max pooling unit to determine the second salient local feature of the second nonlinear feature image; concatenating the second global context information with the second salient local feature to obtain the second feature weight information; and calling the second activation function and the first convolution scale, and weighting the second nonlinear feature image according to the second feature weight information to obtain the second feature weight.

[0147] As one implementation, the spatial weight determination unit includes: a second average pooling unit, a second max pooling unit, a second activation function, and a third convolution scale.

[0148] The feature extraction unit 132 is specifically configured to: call the spatial weight determination unit to determine the first spatial weight of each feature of the first feature weight in the image region of the first nonlinear feature image using a second attention mechanism; and call the second attention mechanism to determine the second spatial weight of each feature of the second feature weight in the image region of the second nonlinear feature image, including: calling the second average pooling unit to determine the first spatial average value of each feature of the first feature weight; and calling the second max pooling unit to determine the first spatial maximum value of each feature of the first feature weight; concatenating the first spatial average value and the first spatial maximum value to obtain the first spatial weight information; and calling the second activation function. The system uses a third convolution scale and, based on the first spatial weight information, further weights each feature of the first feature weight in the image region of the first nonlinear feature image to obtain the first spatial weight. It then calls a second average pooling unit to determine the second spatial average value of each feature of the second feature weight; and calls a second max pooling unit to determine the second spatial maximum value of each feature of the second feature weight. The second spatial average value and the second spatial maximum value are concatenated to obtain the second spatial weight information. Finally, the system calls a second activation function and a third convolution scale, and, based on the second spatial weight information, further weights each feature of the second feature weight in the image region of the second nonlinear feature image to obtain the second spatial weight.

[0149] As one implementation, the feature reconstruction module includes a first reconstruction residual block, a second reconstruction residual block, a third reconstruction residual block, a fourth reconstruction residual block, a fifth reconstruction residual block, and a third activation function; the first reconstruction residual block, the second reconstruction residual block, the third reconstruction residual block, the fourth reconstruction residual block, and the fifth reconstruction residual block have the same structure as the first residual block, the second residual block, and the third residual block.

[0150] The image fusion unit 133 is specifically configured to use a feature reconstruction module to reconstruct features from the first target features in the first feature extraction image and the second target features in the second feature extraction image to obtain a feature fusion image. This includes: calling a first reconstruction residual block to fuse the first feature extraction image and the second feature extraction image to obtain a first fused feature image; performing a first reconstruction nonlinear transformation on the first target features and the second target features to obtain first reconstruction residual features; adding each feature in the first fused feature image to the first reconstruction residual features to obtain a first fused image including the first reconstruction features; and calling a second reconstruction residual block to perform a second reconstruction nonlinear transformation on the first reconstruction features to obtain second reconstruction residual features; adding the first reconstruction features to the second reconstruction residual features to obtain a first fused image including the second reconstruction features. The second fused image of the features is obtained by calling the third reconstruction residual block, performing a third reconstruction nonlinear transformation on the second reconstruction features to obtain third reconstruction residual features, and adding the second reconstruction features and the third reconstruction residual features to obtain a third fused image including the third reconstruction features; the fourth reconstruction residual block is called, performing a fourth reconstruction nonlinear transformation on the third reconstruction features to obtain fourth reconstruction residual features, and adding the third reconstruction features and the fourth reconstruction residual features to obtain a fourth fused image including the fourth reconstruction features; the fifth reconstruction residual block is called, performing a fifth reconstruction nonlinear transformation on the fourth reconstruction features to obtain fifth reconstruction residual features, and adding the fourth reconstruction features and the fifth reconstruction residual features to obtain a fifth fused image including the fifth reconstruction features; the third activation function is called to adjust the scale range of the fifth fused image to obtain a feature fused image.

[0151] Figure 14 This is a schematic diagram of a prediction device provided in this application. Figure 14 The prediction device 60 may include at least one processor 601 and a memory 603 for storing processor-executable instructions. The processor 601 is configured to execute the instructions in the memory 603 to implement the image fusion method described in the following embodiments.

[0152] In addition, the prediction device 60 may also include a communication bus 602, at least one communication interface 604, an input device 606, and an output device 605.

[0153] The processor 601 may be a processor (central processing unit, CPU), a microprocessor unit, an ASIC, or one or more integrated circuits for controlling the execution of the program of the present application.

[0154] The communication bus 602 may include a path for transmitting information between the aforementioned components.

[0155] Communication interface 604 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0156] Input device 606 is used to receive input signals and output device 605 is used to output signals.

[0157] The memory 603 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory may exist independently and be connected to the processing unit via a bus. The memory may also be integrated with the processing unit.

[0158] The memory 603 stores instructions for executing the scheme of this application, and the processor 601 controls the execution. The processor 601 executes the instructions stored in the memory 603 to realize the functions of the method of this application.

[0159] In a specific implementation, as one embodiment, the processor 601 may include one or more CPUs, for example... Figure 14 CPU0 and CPU1 in the CPU.

[0160] In a specific implementation, as one example, the prediction device 60 may include multiple processors, such as... Figure 14 Processors 601 and 607 are described herein. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0161] The predictive device, such as Figure 14The diagram includes a processor 601 and a memory 603 for storing executable instructions of the processor 601; wherein the processor 601 is configured to execute the executable instructions to implement the image fusion method as described in any of the possible embodiments above. And it can achieve the same technical effect, so to avoid repetition, it will not be described again here.

[0162] This application also provides a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of a control device or control apparatus, the control device or control apparatus is able to perform the image fusion method as described in any of the possible embodiments above. And it can achieve the same technical effect; to avoid repetition, it will not be described again here.

[0163] This application also provides a computer program product, including a computer program or instructions, which are executed by a processor as an image fusion method according to any of the possible implementations described above. And it can achieve the same technical effect; to avoid repetition, it will not be described again here.

[0164] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0165] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. An image fusion method, characterized in that, The method includes: Acquire visible light and infrared images of the target object; A feature extraction module is used to extract features from the first target features of the visible light image and the second target features of the infrared image, respectively, to obtain a first feature extraction image and a second feature extraction image. The feature extraction module includes a first unit, a second unit, and a third unit. The first unit is used to extract multi-scale features; the second unit is used to extract nonlinear features; and the third unit is used to perform feature extraction according to an attention mechanism. A feature reconstruction module is used to reconstruct the first target feature in the first feature extraction image and the second target feature in the second feature extraction image to obtain a feature fusion image.

2. The image fusion method according to claim 1, characterized in that, The feature extraction module extracts features from the first target feature in the visible light image and the second target feature in the infrared image, respectively, to obtain a first feature extraction image and a second feature extraction image, including: The first unit is invoked to extract multi-scale features from the visible light image and the infrared image respectively, to obtain a first multi-scale feature image of the visible light image and a second multi-scale feature image corresponding to the infrared image; The second unit is invoked to extract the nonlinear features of the first multi-scale feature image and the second multi-scale feature image respectively, to obtain the first nonlinear feature image corresponding to the first multi-scale feature image and the second nonlinear feature image corresponding to the second multi-scale feature image; The third unit is invoked, and the attention mechanism is used to extract the first target feature of the first nonlinear feature image and the second target feature of the second nonlinear feature image, respectively, to obtain the first feature extraction image and the second feature extraction image.

3. The image fusion method according to claim 2, characterized in that, The first unit includes two first convolution kernels and three convolution kernels with different dilatations; the first convolution kernel is the first convolution scale; The step of calling the first unit to extract multi-scale features from the visible light image and the infrared image respectively, to obtain a first multi-scale feature image of the visible light image and a second multi-scale feature image corresponding to the infrared image, includes: One of the convolutional kernels and the three convolutional kernels with different dilatations are connected in parallel to form the first sub-unit; The first sub-unit is invoked to extract feature information of multiple different scales from the visible light image and the infrared image, respectively, to obtain multiple first sub-feature images and second sub-feature images containing feature information of different scales; Another first convolutional kernel is called to concatenate the features in the first sub-feature image and the features in the second sub-feature image, respectively, to obtain the first multi-scale feature image and the second multi-scale feature image.

4. The image fusion method according to claim 3, characterized in that, The second unit includes a first residual block, a second residual block, and a third residual block; each of the first residual block, the second residual block, and the third residual block includes a second convolution kernel, a third convolution kernel, two first convolution kernels, and a first activation function; the second convolution kernel includes a first activation function and a first convolution scale; the third convolution kernel includes a first activation function and a second convolution scale; The step of calling the second unit to extract nonlinear features from the first multi-scale feature image and the second multi-scale feature image respectively, to obtain a first nonlinear feature image corresponding to the first multi-scale feature image and a second nonlinear feature image corresponding to the second multi-scale feature image, includes: The first residual block is invoked to perform a first feature nonlinear transformation on the first multi-scale feature image to obtain a first visible light transformation function; the first feature value mapped from the first multi-scale feature image is added to the first visible light transformation function to obtain a first visible light residual function; and the first feature nonlinear transformation is performed on the second multi-scale feature image to obtain a first infrared transformation function; the second feature value mapped from the second multi-scale feature image is added to the first infrared transformation function to obtain a first infrared residual function. The second residual block is invoked to perform a second characteristic nonlinear transformation on the first visible light residual function to obtain a second visible light transformation function; the first visible light residual function is added to the second visible light transformation function to obtain a second visible light residual function; and the first infrared residual function is performed to perform a second characteristic nonlinear transformation on the first infrared residual function to obtain a second infrared transformation function; the first infrared residual function is added to the second infrared transformation function to obtain a second infrared residual function. The third residual block is invoked to perform a third characteristic nonlinear transformation on the second visible light residual function to obtain a third visible light transformation function; the second visible light residual function is added to the third visible light transformation function to obtain a third visible light residual function, and a first nonlinear feature image corresponding to the third visible light residual function is obtained; and the second infrared residual function is performed to perform a third characteristic nonlinear transformation to obtain a third infrared transformation function; the second infrared residual function is added to the third infrared transformation function to obtain a third infrared residual function, and a second nonlinear feature image corresponding to the third infrared residual function is obtained.

5. The image fusion method according to claim 4, characterized in that, The third unit includes a feature weight determination unit and a spatial weight determination unit; the third unit is invoked to extract the first target feature of the first nonlinear feature image and the second target feature of the second nonlinear feature image using the attention mechanism, respectively, to obtain the first feature extraction image and the second feature extraction image. include: The feature weight determination unit is invoked, and the first attention mechanism is used to identify the features in the first nonlinear feature image and the second nonlinear feature image respectively, so as to determine the first feature weight of each feature in the first nonlinear feature image and the second feature weight of each feature in the second nonlinear feature image; The spatial weight determination unit is invoked, and a second attention mechanism is used to determine the first spatial weight of each feature of the first feature weight in the image region of the first nonlinear feature image, and the second attention mechanism is used to determine the second spatial weight of each feature of the second feature weight in the image region of the second nonlinear feature image. Based on the first spatial weight and the first feature weight, the first target feature of the first nonlinear feature image is extracted to obtain the first feature extraction image; Furthermore, based on the second spatial weight and the second feature weight, the second target feature of the second nonlinear feature image is extracted to obtain the second feature extraction image.

6. The image fusion method according to claim 5, characterized in that, The feature weight determination unit includes: a first average pooling unit, a first max pooling unit, a second activation function, and a first convolution scale; The feature weight determination unit employs a first attention mechanism to identify features in the first nonlinear feature image and the second nonlinear feature image, respectively, to determine the first feature weight of each feature in the first nonlinear feature image and the second feature weight of each feature in the second nonlinear feature image, including: The first average pooling unit is invoked to determine the first global context information of the first nonlinear feature image; and the first max pooling unit is invoked to determine the first salient local feature of the first nonlinear feature image; the first global context information and the first salient local feature are concatenated to obtain the first feature weight information; the second activation function and the first convolution scale are invoked to weight the first nonlinear feature image according to the first feature weight information to obtain the first feature weight. In addition, the first average pooling unit is invoked to determine the second global context information of the second nonlinear feature image; and the first max pooling unit is invoked to determine the second salient local feature of the second nonlinear feature image; the second global context information and the second salient local feature are concatenated to obtain the second feature weight information; the second activation function and the first convolution scale are invoked to weight the second nonlinear feature image according to the second feature weight information to obtain the second feature weight.

7. The image fusion method according to claim 6, characterized in that, The spatial weight determination unit includes: a second average pooling unit, a second max pooling unit, a second activation function, and a third convolution scale; The spatial weight determination unit uses a second attention mechanism to determine the first spatial weight of each feature of the first feature weight in the image region of the first nonlinear feature image, and uses a second attention mechanism to determine the second spatial weight of each feature of the second feature weight in the image region of the second nonlinear feature image, including: The second average pooling unit is invoked to determine the first spatial average value of each feature of the first feature weight; and the second max pooling unit is invoked to determine the first spatial maximum value of each feature of the first feature weight; the first spatial average value and the first spatial maximum value are concatenated to obtain the first spatial weight information; the second activation function and the third convolution scale are invoked to re-weight each feature of the first feature weight in the image region of the first nonlinear feature image according to the first spatial weight information to obtain the first spatial weight. Furthermore, the second average pooling unit is invoked to determine the second spatial average value of each feature of the second feature weight; and the second max pooling unit is invoked to determine the second spatial maximum value of each feature of the second feature weight; the second spatial average value and the second spatial maximum value are concatenated to obtain the second spatial weight information; the second activation function and the third convolution scale are invoked to reweight each feature of the second feature weight in the image region of the second nonlinear feature image according to the second spatial weight information to obtain the second spatial weight.

8. The image fusion method according to claim 1, characterized in that, The feature reconstruction module includes a first reconstruction residual block, a second reconstruction residual block, a third reconstruction residual block, a fourth reconstruction residual block, a fifth reconstruction residual block, and a third activation function; the first reconstruction residual block, the second reconstruction residual block, the third reconstruction residual block, the fourth reconstruction residual block, and the fifth reconstruction residual block have the same structure as the first residual block, the second residual block, and the third residual block; The feature reconstruction module reconstructs features from the first target feature in the first feature extraction image and the second target feature in the second feature extraction image. The resulting feature-fused image includes: The first reconstruction residual block is invoked to perform image fusion on the first feature extraction image and the second feature extraction image to obtain a first fused feature image; and a first reconstruction nonlinear transformation is performed on the first target feature and the second target feature to obtain a first reconstruction residual feature. Each feature in the first fused feature image is added to the first reconstruction residual feature to obtain a first fused image including the first reconstruction feature. The second reconstruction residual block is invoked to perform a second reconstruction nonlinear transformation on the first reconstruction feature to obtain a second reconstruction residual feature. The first reconstruction feature and the second reconstruction residual feature are added together to obtain a second fused image including the second reconstruction feature. The third reconstruction residual block is invoked to perform a third reconstruction nonlinear transformation on the second reconstruction feature to obtain the third reconstruction residual feature. The second reconstruction feature and the third reconstruction residual feature are added together to obtain a third fused image including the third reconstruction feature. The fourth reconstruction residual block is invoked to perform a fourth reconstruction nonlinear transformation on the third reconstruction feature to obtain the fourth reconstruction residual feature. The third reconstruction feature and the fourth reconstruction residual feature are added together to obtain a fourth fused image including the fourth reconstruction feature. The fifth reconstruction residual block is invoked to perform a fifth reconstruction nonlinear transformation on the fourth reconstruction feature to obtain the fifth reconstruction residual feature. The fourth reconstruction feature and the fifth reconstruction residual feature are added together to obtain the fifth fused image including the fifth reconstruction feature. The third activation function is invoked to adjust the scale range of the fifth fused image, thereby obtaining the feature fused image.

9. The image fusion method according to claim 1, characterized in that, The method further includes: Multiple sets of images are acquired and preprocessed; an image MSRS dataset is constructed based on the preprocessed image data; the image sets represent the visible light image and the infrared image of the same target object; Based on the feature extraction module and the feature reconstruction module, an image fusion network model is constructed; Based on the image MSRS dataset, the image fusion network model is trained until the loss function converges, thus completing the model training. A set of visible light images and infrared images are input into the image fusion network model, and the target feature fusion image is output.

10. An image fusion system, characterized in that, The image fusion system includes a feature extraction module and a feature reconstruction module; the image fusion system is configured to perform the image fusion method as described in any one of claims 1-9.

Images