Image processing method, apparatus, device, storage medium and program product

Abstract

The application relates to an image processing method, device, equipment, storage medium and program product, and relates to the technical field of image processing. The method comprises the following steps: extracting a first feature from a visible light image and extracting a second feature from an infrared image, using a spatial domain processing branch to decouple to obtain the features of the source image in different frequency bands in the spatial domain, using a frequency domain processing branch to decouple to obtain the features of the source image in different frequency bands in the frequency domain, and then reconstructing a fusion image using a decoder based on the frequency bands of the features after fusion. Based on spatial domain decoupling and frequency domain decoupling, local context information modeling and global context information modeling of the fusion image can be realized, the representation capability of complementary features between different modal images is improved, and thus the quality of the fusion image is improved. In addition, the information in different frequency bands can be decoupled out through feature decoupling, the contributions of image background and detail information to the fusion task can be further explained, and the fusion strategy is more targeted and interpretable.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically to an image processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] In the field of image processing, the fusion of visible light and infrared images is a major research focus in image fusion tasks. Visible light images contain detailed information such as texture, while infrared images contain relatively obvious infrared radiation information. Fusing the effective information from these two source images can improve image quality.

[0003] Currently, image fusion methods are relatively simple in extracting features from source images, but they do not fully mine and utilize the information in the source images, which easily leads to the omission of effective information and results in low quality of the fused image. Summary of the Invention

[0004] One of the objectives of this invention is to provide an image processing method, apparatus, device, storage medium, and program product to solve the problem of low image quality after image fusion.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] An image processing method, the method comprising:

[0007] Extract a first feature from a visible light image and extract a second feature from the corresponding infrared image of the visible light image;

[0008] By decoupling the first and second features through the spatial domain processing branch, the features of the visible light image in different frequency bands of the spatial domain and the features of the infrared image in different frequency bands of the spatial domain are obtained.

[0009] By decoupling the first and second features through frequency domain processing branches, the features of visible light images in different frequency bands and infrared images in different frequency bands in the frequency domain are obtained.

[0010] The features of visible light images and infrared images in the same frequency band are fused, and the fusion result is reconstructed using a decoder to obtain the reconstructed image.

[0011] Furthermore, the first and second features are decoupled using the gradient residual algorithm to obtain the features of the visible light image in different frequency bands of the spatial domain and the features of the infrared image in different frequency bands of the spatial domain, including:

[0012] A downsampling operation is performed on the feature to be decoupled, and an upsampling operation is performed on the downsampling result using interpolation to obtain an upsampling result with the same resolution as the feature to be decoupled, which is either the first feature or the second feature.

[0013] Calculate the difference between the feature to be decoupled and the upsampling result, and determine the high-frequency features in the spatial domain based on the difference and the preset gradient operator;

[0014] Low-frequency features of the features to be decoupled in the spatial domain are extracted using a low-pass filter or an attention-based image processing model.

[0015] Furthermore, by decoupling the first and second features using a frequency domain mask filter, features of the visible light image in different frequency bands and features of the infrared image in different frequency bands are obtained, including:

[0016] Perform a Fourier transform on the feature to be decoupled to obtain its frequency domain representation. The feature to be decoupled is either the first feature or the second feature.

[0017] By performing a masking operation on the frequency domain representation of the feature to be decoupled using a preset high-frequency mask, the high-frequency features of the feature to be decoupled in the frequency domain are obtained.

[0018] By performing a masking operation on the frequency domain representation of the feature to be decoupled using a preset low-frequency mask, the low-frequency features of the feature to be decoupled in the frequency domain are obtained.

[0019] Furthermore, features from different frequency bands include high-frequency features and low-frequency features. Features from visible light images and infrared images in the same frequency band are fused, including:

[0020] High-frequency features in the spatial domain are combined with those in the visible light and infrared images to obtain high-frequency combined features in the spatial domain, and low-frequency features in the spatial domain are combined with those in the visible light and infrared images to obtain low-frequency combined features in the spatial domain.

[0021] The high-frequency features of visible light images and infrared images in the frequency domain are combined to obtain high-frequency combined features in the frequency domain, and the low-frequency features of visible light images and infrared images in the frequency domain are combined to obtain low-frequency combined features in the frequency domain.

[0022] A reversible neural network is used to fuse the high-frequency combination features in the spatial domain and the high-frequency combination features in the frequency domain, and the fused features are obtained as high-frequency comprehensive features.

[0023] A reversible neural network is used to fuse the low-frequency combination features in the spatial domain and the low-frequency combination features in the frequency domain, and the fused low-frequency comprehensive features are obtained.

[0024] Furthermore, a reversible neural network is used to fuse high-frequency combination features in the spatial domain and high-frequency combination features in the frequency domain, including:

[0025] The high-frequency combined features in the spatial domain are adjusted by a scaling transformation function, and the adjustment result is used to calculate the Hadamard product with the high-frequency combined features in the frequency domain. The calculation result is used as the first fused feature.

[0026] And / or,

[0027] The high-frequency combined features in the frequency domain are adjusted by a scaling transformation function, and the adjustment result is used to calculate the Hadamard product with the high-frequency combined features in the spatial domain. The result is then used as the second fused feature.

[0028] Furthermore, the encoder is obtained based on the encoding module of the image restoration model Restormer, and the decoder is obtained based on the decoding module of Restormer.

[0029] An image processing apparatus, comprising:

[0030] An extraction module is used to extract a first feature from a visible light image and a second feature from an infrared image corresponding to the visible light image;

[0031] The spatial domain decoupling module is used to decouple the first feature and the second feature respectively through the gradient residual algorithm to obtain the features of the visible light image in different frequency bands in the spatial domain and the features of the infrared image in different frequency bands in the spatial domain.

[0032] The frequency domain decoupling module is used to decouple the first feature and the second feature respectively through a frequency domain mask filter to obtain the features of the visible light image in different frequency bands and the features of the infrared image in different frequency bands.

[0033] The reconstruction module is used to fuse features of visible light and infrared images in the same frequency band, and then use a decoder to reconstruct the fusion result to obtain the reconstructed image.

[0034] An electronic device includes: a processor, and a memory communicatively connected to the processor;

[0035] The memory stores computer-executed instructions;

[0036] The processor executes computer execution instructions stored in the memory to implement the image processing method as described in any of the above.

[0037] A computer-readable storage medium includes: computer-executable instructions stored in the computer-readable storage medium, which, when executed by a processor, are used to implement the image processing method as described in any of the preceding claims.

[0038] A computer program product includes a computer program that, when executed by a processor, implements the image processing method as described in any of the preceding claims.

[0039] The beneficial effects of this invention are:

[0040] (1) Based on the two dimensions of spatial domain decoupling and frequency domain decoupling, the local modeling attributes of image vision tasks in the spatial domain and the global modeling attributes in the frequency domain can be used to realize the modeling of local context information and global context information of fused images, thereby improving the representation ability of complementary features between different modal images, thereby improving the robustness and generalization of image fusion and the quality of fused images.

[0041] (2) By decoupling information from different frequency bands through features, we can further explain the contribution of image background and detail information to the fusion task, making the fusion strategy more targeted and interpretable. Attached Figure Description

[0042] Figure 1 A flowchart illustrating an image processing method provided for an exemplary embodiment of the present invention;

[0043] Figure 2 A schematic diagram illustrating the principle of image fusion provided for an exemplary embodiment of the present invention;

[0044] Figure 3 A schematic diagram of a spatial domain feature decoupling process provided for an exemplary embodiment of the present invention;

[0045] Figure 4 A schematic diagram of a frequency domain feature decoupling process provided for an exemplary embodiment of the present invention;

[0046] Figure 5 A flowchart illustrating the fusion of features in the same frequency band is provided for an exemplary embodiment of the present invention.

[0047] Figure 6 A schematic diagram of the structure of an image processing apparatus provided for an exemplary embodiment of the present invention;

[0048] Figure 7 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present invention.

[0049] The accompanying drawings have illustrated specific embodiments of the invention, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the invention in any way, but rather to illustrate the concept of the invention to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0050] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0051] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0052] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0053] The terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, product, or apparatus. Without further limitation, the presence of other identical or equivalent elements in the process, method, product, or apparatus that includes said elements is not excluded. For example, the use of terms such as "first," "second," etc., is to denote names and does not indicate any particular order.

[0054] Infrared images primarily acquire images by detecting the infrared radiation emitted by objects, enabling them to provide high-level target detection and identification capabilities at night or in adverse weather conditions. However, infrared images typically have lower resolution and poorer texture information. In contrast, visible light images can provide rich color and texture information, making scenes appear more realistic and vivid. However, their image quality degrades significantly at night or in adverse weather conditions.

[0055] The purpose of infrared and visible light image fusion is to effectively combine the complementary information from infrared and visible light images to generate a new image that contains both significant infrared thermal radiation information and detailed visible light texture information. By combining the complementary advantages of these two types of images, image fusion technology can produce images containing prominent targets and rich texture details.

[0056] Currently, deep learning technology is increasingly being applied to solve image fusion problems. Fusion methods can include feature extraction, feature fusion, and image reconstruction. However, deep learning-based image fusion often uses a shared feature extractor, failing to consider the complementarity and redundancy of multimodal features. This results in strong feature coupling, insufficient mining and utilization of information in the image, and a tendency to miss effective information, leading to a decline in the quality of the fused image. Furthermore, the fusion strategy is often manually designed based on coupled features, lacking good interpretability.

[0057] Based on this, this invention proposes a technical concept that extracts features from visible light and infrared images separately, and designs two decoupling branches: one in the spatial domain and one in the frequency domain. In the spatial domain decoupling branch, the extracted features are further decoupled to obtain features in different frequency bands in the spatial domain. Similarly, in the frequency domain decoupling branch, the extracted features are further decoupled to obtain features in different frequency bands. Then, features are fused based on their frequency bands. Features belonging to the same frequency band are complementary and interactively fused, and then the image is reconstructed using a decoder. Based on the spatial and frequency domain decoupling branches, the local modeling attributes in the spatial domain and the global modeling attributes in the frequency domain of image vision tasks can be utilized to model the local and global contextual information of the fused image. This improves the representation ability of complementary features between different modalities, thereby enhancing the robustness and generalization of image fusion and the quality of the fused image. Furthermore, by decoupling the information from different frequency bands, the contribution of image background and detail information to the fusion task can be further explained, making the fusion strategy more targeted and interpretable.

[0058] The application scenarios mentioned above are only partial examples. Those skilled in the art can expand the applications based on specific testing needs and scenarios. The embodiments of this application do not impose specific limitations in this regard. The following references Figures 1 to 6 This describes a test method according to an exemplary embodiment of the present invention.

[0059] Figure 1 This is a schematic flowchart illustrating an image processing method provided as an exemplary embodiment of the present invention. For example... Figure 1 As shown, the method may include:

[0060] Step S101: Extract a first feature from the visible light image and extract a second feature from the infrared image corresponding to the visible light image.

[0061] Visible light images and infrared images can represent the same scene, and can be referred to as source images.

[0062] For example, visible light images and infrared images can be respectively input into a pre-trained neural network, which may include multiple convolutional layers for extracting image features.

[0063] In this embodiment of the invention, an image processing model based on an autoencoder architecture can be used to perform preliminary feature extraction from the source image and reconstruction of the fused image. An autoencoder, or encoder-decoder, involves the encoder extracting features from the input image through operations such as convolution to obtain feature maps of different forms; the decoder then performs image restoration and reconstruction based on the feature maps provided by the encoder or other modules to obtain the reconstructed image.

[0064] For example, a visible light image and an infrared image can be input into the same encoder, with the encoder output corresponding to the visible light image being denoted as the first feature and the output corresponding to the infrared image being denoted as the second feature.

[0065] Step S102: Decouple the first feature and the second feature by the spatial domain processing branch to obtain the features of the visible light image in different frequency bands in the spatial domain and the features of the infrared image in different frequency bands in the spatial domain.

[0066] The features in different frequency bands can include high-frequency features and low-frequency features. High-frequency features can reflect local details of the image (such as edges and textures), while low-frequency features can reflect global information of the image (such as the background).

[0067] In this embodiment of the invention, a high-frequency filtering module (HFM) can be used as a basic unit, combined with a gradient residual structure design, to obtain high-frequency features of visible light images and infrared images. For example, the high-frequency filtering module can be used to extract high-frequency features from the source image, and the extracted high-frequency features can be used as residuals. Gradient information is then used to construct residual connections. By transmitting information to deeper layers of the network through the gradient residual structure, the retention and transmission of high-frequency information can be enhanced at different levels, helping the network to better learn high-frequency features.

[0068] For example, a lightweight Transformer can be used to decouple the low-frequency representations in the first or second feature to obtain the low-frequency features of the source image. Here, the Transformer is a deep learning model based on an attention mechanism.

[0069] Step S103: Decouple the first feature and the second feature by frequency domain processing branch to obtain the features of the visible light image in different frequency bands and the features of the infrared image in different frequency bands.

[0070] The frequency domain processing branch may include a frequency domain mask filter. The frequency domain mask filter may contain a frequency mask with radius r, used to separate features at different frequencies. Optionally, the size of r is 10 pixels.

[0071] Taking the decoupling of the first feature as an example, the frequency domain mask filter can first transform the first feature to the frequency domain using Discrete Fourier Transform, and then use a frequency mask with radius r to decouple the frequency domain transformation result of the first feature, thereby obtaining the high-frequency and low-frequency features of the visible light image in the frequency domain. Correspondingly, the second feature can also be decoupled in this way to obtain the high-frequency and low-frequency features of the infrared image in the frequency domain.

[0072] Step S104: The features of the visible light image and the infrared image in the same frequency band are fused, and the fusion result is reconstructed using a decoder to obtain the reconstructed image.

[0073] In this embodiment of the invention, this step aims to perform complementary interactive fusion of features from different sources according to frequency bands, which can be high-frequency or low-frequency.

[0074] Taking high frequency as an example, the high-frequency features of a visible light image can include its high-frequency features in the spatial domain and its high-frequency features in the frequency domain. Similarly, the high-frequency features of an infrared image can include its high-frequency features in the spatial domain and its high-frequency features in the frequency domain. These high-frequency features can be fused together through weighted combination or matrix operations to obtain the unified high-frequency features of both visible light and infrared images.

[0075] It should be noted that the fusion of high-frequency features can be performed by first performing preliminary fusion on the features of the same source image in the spatial domain and frequency domain, and then performing final fusion on the preliminary fusion results of different source images. Alternatively, it can be performed by first performing preliminary fusion on the features of different source images in the same domain (such as the spatial domain), and then performing final fusion on the preliminary fusion results of different domains. Alternatively, all high-frequency features can be fused at once. This embodiment does not impose any restrictions on this.

[0076] In the above embodiments, by extracting a first feature from a visible light image and a second feature from an infrared image, the gradient residual algorithm can be used to decouple and obtain the features of the source image in different frequency bands in the spatial domain. Alternatively, a frequency domain mask filter can be used to decouple and obtain the features of the source image in different frequency bands in the frequency domain. After fusing based on the frequency bands of these features, a decoder can be used to reconstruct the fused image. Based on spatial and frequency domain decoupling, the local modeling attributes in the spatial domain and the global modeling attributes in the frequency domain of image vision tasks can be utilized to model the local and global contextual information of the fused image. This improves the representation ability of complementary features between different modalities, thereby enhancing the robustness and generalization of image fusion and the quality of the fused image. Furthermore, by decoupling information from different frequency bands through features, the contribution of image background and detail information to the fusion task can be further explained, making the fusion strategy more targeted and interpretable.

[0077] Figure 2 This is a schematic diagram illustrating the principle of image fusion as an example of the present invention. The following is in conjunction with... Figure 2 The present invention will be further illustrated by specific embodiments.

[0078] like Figure 2 As shown, the image processing method in this invention can use an encoder for feature extraction and a decoder for image reconstruction. It also includes two decoupled branches: spatial domain and frequency domain. By fusing the features processed by the two decoupled branches, high-frequency comprehensive features and low-frequency comprehensive features can be obtained.

[0079] In some possible implementations, the encoder can contain multiple coding layers, which can consist of a series of operations such as convolution, activation, and / or pooling. After multiple operations, the encoder can extract feature maps as shallow features of the image.

[0080] In some possible implementations, the decoder can contain multiple decoding layers, and the structure of the decoder can be symmetrical with that of the encoder.

[0081] In one embodiment, the encoder is obtained based on the encoding module of the Restormer image restoration model, and the decoder is obtained based on the decoding module of Restormer.

[0082] Restormer is a deep learning model for image restoration tasks, and it is a variant of the Transformer architecture. In this embodiment of the invention, Restormer blocks can be used as basic units to form encoders and decoders.

[0083] In one embodiment, such as Figure 3As shown, by decoupling the first and second features through spatial domain processing branches, features of the visible light image in different frequency bands of the spatial domain and features of the infrared image in different frequency bands of the spatial domain are obtained, which may include:

[0084] Step S301: Perform downsampling on the feature to be decoupled, and then perform upsampling on the downsampling result using interpolation to obtain an upsampling result with the same resolution as the feature to be decoupled.

[0085] The feature to be decoupled is either the first feature or the second feature.

[0086] Step S302: Calculate the difference between the feature to be decoupled and the upsampling result, and determine the high-frequency feature in the spatial domain based on the difference and the preset gradient operator.

[0087] For example, downsampling and upsampling can be achieved through the high-frequency filtering module HFM. Downsampling can be achieved through convolution operations. By setting the convolution kernel size, a downsampling result with a resolution of one-third of the feature to be decoupled can be obtained. Upsampling can be achieved through bilinear interpolation. By interpolating the downsampling result, an upsampling result with the same resolution as the feature to be decoupled can be obtained. The residual is calculated based on the feature to be decoupled and the upsampling result as the difference between the two. This process can be expressed as the following formula (1).

[0088] HFM(F) = F - Up(Down(F)) (1)

[0089] In the above formula, This can represent the first or second feature to be decoupled. Down(·) indicates a downsampling operation, and Up(·) indicates an upsampling operation. The left side of the equals sign can represent the residual calculation result of HFM with the feature to be decoupled F as input. The subtraction operation on the right side of the equals sign can be a matrix-by-matrix subtraction. Since the feature to be decoupled F and the upsampling result have the same resolution and the same number of pixels, the pixel values ​​at the same coordinates can be subtracted after aligning their coordinates to obtain the residual.

[0090] For example, based on the difference between the feature to be decoupled and the upsampling result, the Sobel gradient operator and the residual connection operation can be used to obtain the final high-frequency features. This process can be expressed as the following formula (2).

[0091] (2)

[0092] In the above formula, HFMs(·) can represent multiple cascaded HFMs(·) and convolutional downsampling layers, and δ(·) can represent the Sobel gradient operator.

[0093] Step S303: Extract the low-frequency features of the features to be decoupled in the spatial domain using a low-pass filter or an image processing model based on an attention mechanism.

[0094] For example, the features to be decoupled can be smoothed using a filter. For instance, the kernel size can be set, and low-frequency components in the first or second feature can be obtained directly through Gaussian blur. The larger the kernel, the stronger the low-frequency retention.

[0095] For example, a lightweight model, Lite Transformer, can be used to process the features to be decoupled. Sparse global attention (such as sliding window attention of Longformer) or linear attention (such as Performer) can be used to capture global dependencies across regions to obtain low-frequency features of the image.

[0096] In the above embodiments, by decoupling the components of different frequency bands in the spatial domain from the first feature and the second feature, the edge, texture and other details, as well as the background and other global information in the visible light image and infrared image can be characterized from the perspective of the spatial domain, further explaining the contribution of image background and detail information to the fusion task.

[0097] In one embodiment, such as Figure 4 As shown, by decoupling the first and second features through frequency domain processing branches, features of the visible light image in different frequency bands and features of the infrared image in different frequency bands are obtained, which may include:

[0098] Step S401: Perform a Fourier transform on the feature to be decoupled to obtain the frequency domain representation of the feature to be decoupled.

[0099] The feature to be decoupled is either the first feature or the second feature.

[0100] Step S402: Perform a masking operation on the frequency domain representation of the feature to be decoupled using a preset high-frequency mask to obtain the high-frequency features of the feature to be decoupled in the frequency domain.

[0101] Step S403: Perform a masking operation on the frequency domain representation of the feature to be decoupled using a preset low-frequency mask to obtain the low-frequency feature of the feature to be decoupled in the frequency domain.

[0102] For example, the first feature to be decoupled can be transformed to the frequency domain using the following formula (3):

[0103] (3)

[0104] Where F can represent the feature to be decoupled. It can represent the Fourier transform function. F fre It can represent the frequency domain representation of the feature to be decoupled, that is, the Fourier transform result of the feature F to be decoupled.

[0105] Based on the previous step, the high and low frequency information of the source image can be obtained by decoupling using the frequency mask operation according to the following formulas (4) and (5):

[0106] (4)

[0107] (5)

[0108] In the above formula, Mask high (·) can represent high-frequency masking operations, Mask low (·) can represent low-frequency masking operations. This can represent high-frequency features obtained through high-frequency masking operations. It can represent low-frequency features obtained through high-frequency masking operations.

[0109] In some possible implementations, multiple high-frequency masks with different thresholds can be set, the Fourier transform result of the feature to be decoupled can be masked with each high-frequency mask respectively, and the results of multiple mask operations can be weighted and combined to obtain the final high-frequency feature.

[0110] The process for obtaining low-frequency features can be referred to the process for obtaining high-frequency features described above, and will not be repeated here.

[0111] High-frequency features in the spatial domain (such as gradients and edges) only reflect local abrupt changes and may ignore global periodic or directional structures. Furthermore, high-frequency features in the spatial domain are easily affected by noise, leading to artifacts in the fusion results.

[0112] In the above embodiments, Fourier transform and frequency domain masking operations can be used to obtain high and low frequency information of visible light and infrared images in the frequency domain. This allows image fusion to utilize features from both the spatial and frequency domains. Combining these two dimensions preserves the geometric location information in the spatial domain while utilizing the multi-scale energy distribution in the frequency domain, achieving a more comprehensive feature representation. Furthermore, the high-frequency edges in the spatial domain and their corresponding high-frequency energies in the frequency domain can be cross-validated to eliminate abnormal noise points, thereby improving the quality of image fusion. In addition, the information contribution of different frequency bands can be intuitively analyzed through spectrograms, enhancing the interpretability of image fusion and facilitating the debugging of fusion strategies.

[0113] In one embodiment, such as Figure 5 As shown, fusing features from visible light and infrared images in the same frequency band can include:

[0114] Step S501: Combine the high-frequency features of the visible light image and the infrared image in the spatial domain to obtain the high-frequency combined features in the spatial domain, and combine the low-frequency features of the visible light image and the infrared image in the spatial domain to obtain the low-frequency combined features in the spatial domain.

[0115] For example, the high-frequency features of the visible light image and the infrared image in the spatial domain can be added together according to the following formula (6) to obtain the high-frequency combination feature L0 in the spatial domain, and the low-frequency features of the visible light image and the infrared image in the spatial domain can be added together according to the following formula (7) to obtain the low-frequency combination feature L0' in the spatial domain.

[0116] (6)

[0117] (7)

[0118] In the above formula, It can represent the high-frequency characteristics of infrared images in the spatial domain. It can represent the high-frequency characteristics of visible light images in the spatial domain. It can represent the low-frequency characteristics of infrared images in the spatial domain. It can represent the low-frequency characteristics of visible light images in the spatial domain.

[0119] Step S502: Combine the high-frequency features of the visible light image and the infrared image in the frequency domain to obtain high-frequency combined features in the frequency domain, and combine the low-frequency features of the visible light image and the infrared image in the frequency domain to obtain low-frequency combined features in the frequency domain.

[0120] In this embodiment of the invention, before combining the high-frequency features of the visible light image and the infrared image in the frequency domain, the high-frequency features in the frequency domain can be converted into a spatial domain representation according to the following formulas (8) and (9).

[0121] (8)

[0122] (9)

[0123] In the above formula, It can represent the inverse Fourier transform. This can be represented by the inverse Fourier transform, which extracts high-frequency features from the frequency domain. The result after conversion to the spatial domain, This can be represented by the inverse Fourier transform, which extracts high-frequency features from the frequency domain. The result after conversion to the spatial domain.

[0124] Based on this, the high-frequency features of the visible light image and the infrared image in the frequency domain can be added together according to the following formula (10) to obtain the high-frequency combination feature G0 in the frequency domain, and the low-frequency features of the visible light image and the infrared image in the frequency domain can be added together according to the following formula (11) to obtain the low-frequency combination feature G0' in the frequency domain.

[0125] (10)

[0126] (11)

[0127] In the above formula, It can represent the high-frequency characteristics of infrared images in the frequency domain. It can represent the high-frequency characteristics of visible light images in the frequency domain. It can represent the low-frequency characteristics of infrared images in the frequency domain. It can represent the low-frequency characteristics of visible light images in the frequency domain.

[0128] Step S503: A reversible neural network is used to fuse the high-frequency combination features in the spatial domain and the high-frequency combination features in the frequency domain to obtain the high-frequency comprehensive features.

[0129] In this embodiment of the invention, a dense INN fusion network can be used to fuse high-frequency or low-frequency features from the spatial and frequency domains, achieving complementary interactive fusion of features from different sources. INN (Invertible Neural Networks) are reversible neural networks.

[0130] In some possible implementations, the spatial high-frequency combined features can be adjusted using a scaling transformation function, and the adjustment result can be used to calculate the Hadamard product with the frequency-domain high-frequency combined features. The result is then used as the first fused feature. In this case, the high-frequency integrated feature is the first fused feature L1, and the fusion process can be expressed as formula (12).

[0131] (12)

[0132] Where exp(·) can represent a mathematical exponential function. and Let these represent the scaling function and the transformation function, respectively. It can represent the Hadamard product operation.

[0133] This calculation method allows us to use the spatial high-frequency combination feature L0 as the main component, supplemented by the frequency domain high-frequency combination feature G0, to calculate the high-frequency integrated feature fused under the dominance of the spatial features.

[0134] In some possible implementations, the frequency domain high-frequency combined features can be adjusted using a scaling transformation function, and the adjustment result can be used to calculate the Hadamard product with the spatial domain high-frequency combined features. The result is then used as the second fusion feature. In this case, the high-frequency integrated feature is the second fusion feature G1, and the fusion process can be expressed as formula (13).

[0135] (13)

[0136] Understandable, and For scaling functions with different parameters, and These are transformation functions with different parameters.

[0137] This calculation method allows us to use the high-frequency combination feature G0 in the frequency domain as the main component, supplemented by the high-frequency combination feature L0 in the spatial domain, to calculate the high-frequency integrated feature fused under the dominance of the frequency domain features.

[0138] In some possible implementations, both of the above calculation methods can be used simultaneously. In this case, the high-frequency synthesis features can simultaneously include L1 and G1 as well as any weighted combination of the two.

[0139] Step S504: A reversible neural network is used to fuse the spatial domain low-frequency combination features and the frequency domain low-frequency combination features to obtain low-frequency comprehensive features.

[0140] In this embodiment of the invention, the process of obtaining low-frequency comprehensive features can refer to the process of calculating high-frequency comprehensive features using the above formulas (12) and (13), replacing L0 with L0' and G0 with G0', which will not be repeated here.

[0141] In the above embodiments, the reversible neural network can achieve efficient interaction between features in both the spatial and frequency domains, making full use of image information expressed in different domains and improving image fusion quality.

[0142] Figure 6 This is a schematic diagram of the structure of an image processing apparatus provided for an exemplary embodiment of the present invention. For example... Figure 6 As shown, the image processing apparatus 600 may include:

[0143] Extraction module 601 is used to extract a first feature from a visible light image and extract a second feature from an infrared image corresponding to the visible light image;

[0144] The spatial domain decoupling module 602 is used to decouple the first feature and the second feature respectively through the spatial domain processing branch to obtain the features of the visible light image in different frequency bands in the spatial domain and the features of the infrared image in different frequency bands in the spatial domain.

[0145] The frequency domain decoupling module 603 is used to decouple the first feature and the second feature respectively through frequency domain processing branches to obtain the features of the visible light image in different frequency bands in the frequency domain and the features of the infrared image in different frequency bands in the frequency domain.

[0146] The reconstruction module 604 is used to fuse the features of the visible light image and the infrared image in the same frequency band, and use a decoder to reconstruct the fusion result to obtain the reconstructed image.

[0147] In one embodiment, the spatial decoupling module 602 is further configured to: perform a downsampling operation on the feature to be decoupled, and perform upsampling based on the downsampling result using an interpolation method to obtain an upsampling result with the same resolution as the feature to be decoupled, wherein the feature to be decoupled is the first feature or the second feature; calculate the difference between the feature to be decoupled and the upsampling result, and determine the high-frequency feature in the spatial domain based on the difference and a preset gradient operator; and extract the low-frequency feature in the spatial domain of the feature to be decoupled using a low-pass filter or an image processing model based on an attention mechanism.

[0148] In one embodiment, the frequency domain decoupling module 603 is further configured to: perform a Fourier transform on the feature to be decoupled to obtain a frequency domain representation of the feature to be decoupled, wherein the feature to be decoupled is the first feature or the second feature; perform a masking operation on the frequency domain representation of the feature to be decoupled using a preset high-frequency mask to obtain a high-frequency feature of the feature to be decoupled in the frequency domain; and perform a masking operation on the frequency domain representation of the feature to be decoupled using a preset low-frequency mask to obtain a low-frequency feature of the feature to be decoupled in the frequency domain.

[0149] In one embodiment, the reconstruction module 604 is further configured to: combine the high-frequency features of the visible light image and the infrared image in the spatial domain to obtain a spatial high-frequency combined feature, and combine the low-frequency features of the visible light image and the infrared image in the spatial domain to obtain a spatial low-frequency combined feature; combine the high-frequency features of the visible light image and the infrared image in the frequency domain to obtain a frequency-domain high-frequency combined feature, and combine the low-frequency features of the visible light image and the infrared image in the frequency domain to obtain a frequency-domain low-frequency combined feature; fuse the spatial high-frequency combined feature and the frequency-domain high-frequency combined feature using a reversible neural network to obtain a high-frequency comprehensive feature; and fuse the spatial low-frequency combined feature and the frequency-domain low-frequency combined feature using a reversible neural network to obtain a low-frequency comprehensive feature.

[0150] In one embodiment, the reconstruction module 604 is further configured to: adjust the spatial high-frequency combination features using a scaling transformation function, calculate the Hadamard product between the adjustment result and the frequency high-frequency combination features, and use the calculation result as a first fusion feature; and / or, adjust the frequency high-frequency combination features using a scaling transformation function, calculate the Hadamard product between the adjustment result and the spatial high-frequency combination features, and use the calculation result as a second fusion feature.

[0151] The image processing apparatus provided in this embodiment is used to execute the technical solutions in any of the foregoing method embodiments. Its implementation principle and technical effects are similar, and will not be described again here.

[0152] It should be understood that the above-described device embodiments are merely illustrative, and the device of the present invention can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0153] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of the present invention can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0154] Figure 7 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present invention. For example... Figure 7 As shown, the electronic device 70 includes:

[0155] Processor 71, memory 72, and communication interface 73;

[0156] The memory 72 is used to store the executable instructions of the processor 71; the executable instructions may be computer-executable instructions.

[0157] The processor 71 is configured to execute the technical solutions in any of the foregoing method embodiments by executing the executable instructions.

[0158] Optionally, the memory 72 can be either standalone or integrated with the processor 71.

[0159] Optionally, when the memory 72 is a device independent of the processor 71, the electronic device 70 may further include:

[0160] Bus 74, memory 72 and communication interface 73 are connected to processor 71 through bus 74 and complete communication with each other. Communication interface 73 is used to communicate with other devices.

[0161] Optionally, the communication interface 73 can be implemented using a transceiver. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write databases, and read-only databases). The memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk drive.

[0162] Bus 74 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one line is used in the diagram, but this does not imply that there is only one bus or one type of bus.

[0163] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0164] The electronic device is used to execute the technical solutions in any of the foregoing method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0165] This invention also provides a readable storage medium, which can be a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the technical solution provided in any of the foregoing method embodiments.

[0166] This invention also provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the technical solutions provided in any of the foregoing method embodiments.

[0167] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0168] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0169] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0170] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that, include: Extract a first feature from a visible light image and extract a second feature from an infrared image corresponding to the visible light image; By decoupling the first feature and the second feature through spatial domain processing branches, the features of the visible light image in different frequency bands of the spatial domain and the features of the infrared image in different frequency bands of the spatial domain are obtained. By decoupling the first feature and the second feature through frequency domain processing branches, the features of the visible light image in different frequency bands and the features of the infrared image in different frequency bands are obtained. The visible light image and the infrared image are fused together using features in the same frequency band, and the fusion result is reconstructed using a decoder to obtain the reconstructed image.

2. The image processing method according to claim 1, characterized in that, The step of decoupling the first feature and the second feature through spatial domain processing branches to obtain the features of the visible light image in different frequency bands of the spatial domain and the features of the infrared image in different frequency bands of the spatial domain includes: A downsampling operation is performed on the feature to be decoupled, and an upsampling operation is performed on the downsampling result using interpolation to obtain an upsampling result with the same resolution as the feature to be decoupled, wherein the feature to be decoupled is the first feature or the second feature. Calculate the difference between the feature to be decoupled and the upsampling result, and determine the high-frequency feature in the spatial domain based on the difference and a preset gradient operator; The low-frequency features of the features to be decoupled in the spatial domain are extracted using a low-pass filter or an image processing model based on an attention mechanism.

3. The image processing method according to claim 1 or 2, characterized in that, The step of decoupling the first feature and the second feature through frequency domain processing branches to obtain the features of the visible light image in different frequency bands and the features of the infrared image in different frequency bands includes: Perform a Fourier transform on the feature to be decoupled to obtain its frequency domain representation, wherein the feature to be decoupled is either the first feature or the second feature. By performing a masking operation on the frequency domain representation of the feature to be decoupled using a preset high-frequency mask, the high-frequency features of the feature to be decoupled in the frequency domain are obtained. By performing a masking operation on the frequency domain representation of the feature to be decoupled using a preset low-frequency mask, the low-frequency features of the feature to be decoupled in the frequency domain are obtained.

4. The image processing method according to claim 1 or 2, characterized in that, The features of the different frequency bands include high-frequency features and low-frequency features. The fusion of features from the visible light image and the infrared image in the same frequency band includes: The high-frequency features of the visible light image and the infrared image in the spatial domain are combined to obtain high-frequency combined features in the spatial domain, and the low-frequency features of the visible light image and the infrared image in the spatial domain are combined to obtain low-frequency combined features in the spatial domain. The high-frequency features of the visible light image and the infrared image in the frequency domain are combined to obtain high-frequency combined features in the frequency domain, and the low-frequency features of the visible light image and the infrared image in the frequency domain are combined to obtain low-frequency combined features in the frequency domain. A reversible neural network is used to fuse the spatial high-frequency combination features and the frequency domain high-frequency combination features to obtain high-frequency comprehensive features. A reversible neural network is used to fuse the spatial domain low-frequency combination features and the frequency domain low-frequency combination features to obtain low-frequency comprehensive features.

5. The image processing method according to claim 4, characterized in that, The process of fusing the spatial high-frequency combination features and the frequency high-frequency combination features using a reversible neural network includes: The spatial high-frequency combination features are adjusted by a scaling transformation function, and the adjustment result is used to calculate the Hadamard product with the frequency high-frequency combination features. The calculation result is used as the first fusion feature. And / or, The frequency domain high-frequency combination features are adjusted by a scaling transformation function, and the adjustment result is used to calculate the Hadamard product with the spatial domain high-frequency combination features. The calculation result is used as the second fused feature.

6. The image processing method according to claim 1 or 2, characterized in that, The decoder is based on the decoding module of the Restormer image restoration model.

7. An image processing apparatus, characterized in that, include: An extraction module is used to extract a first feature from a visible light image and a second feature from an infrared image corresponding to the visible light image; The spatial decoupling module is used to decouple the first feature and the second feature respectively through the spatial processing branch to obtain the features of the visible light image in different frequency bands in the spatial domain and the features of the infrared image in different frequency bands in the spatial domain. The frequency domain decoupling module is used to decouple the first feature and the second feature respectively through frequency domain processing branches to obtain the features of the visible light image in different frequency bands in the frequency domain and the features of the infrared image in different frequency bands in the frequency domain. The reconstruction module is used to fuse the features of the visible light image and the infrared image in the same frequency band, and use a decoder to reconstruct the fusion result to obtain the reconstructed image.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 6.

Images