Image fusion network convolution kernel initialization method based on frequency domain feature extraction

Abstract

The invention discloses an image fusion network convolution kernel initialization method based on frequency domain feature extraction, and relates to the technical field of deep learning and computer vision. In order to solve the problems of slow training convergence and limited performance caused by random initialization of a deep network in an infrared and visible light image fusion task, the method comprises the following steps: firstly, carrying out normalization and block preprocessing on a registered source image, and then analyzing frequency domain characteristics of each image block by adopting an SAFD algorithm based on fast Fourier transform; and extracting a complex decomposition coefficient and aggregating real parts into a global feature vector, and finally converting the vector into an airspace convolution kernel through tensor remodeling operation, which is specially used for initializing a bottom layer weight of a corresponding branch of the fusion network. The method provides priori knowledge with physical significance for the network, can significantly accelerate the training process and improve the quality of the final fused image, and can be widely applied to an end-side image fusion system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning and image processing, and particularly relates to an image fusion network convolution kernel initialization method based on frequency domain feature extraction. The method analyzes the essential frequency domain characteristics of an image through Stochastic Adaptive Fourier Decomposition (SAFD), constructs a convolution kernel with explicit physical meaning, and is used for initializing an infrared and visible light image fusion neural network, thereby significantly improving the convergence speed of network training and the quality of the final fusion image. The present application can be widely applied to end-side vision tasks that require rapid deployment and high-precision fusion, such as autonomous driving, security monitoring and remote sensing image analysis. BACKGROUND

[0002] With the rapid development of deep learning technology, the image fusion method based on convolutional neural network has completely surpassed the traditional multi-scale transformation method in performance. However, a frequently overlooked but crucial problem is that the weight initialization strategy, which is the starting point of these high-performance networks, has remained at the level of a decade ago. Although the current mainstream Xavier and He initialization methods solve the problem of training stability to some extent, their essence is to design a general scheme based on the mathematical statistical characteristics of the weight matrix, completely ignoring the prior knowledge specific to the image fusion task.

[0003] This “one-size-fits-all” initialization method has fundamental defects. First, it forces each new model to start exploring a huge parameter space from a completely random starting point, resulting in a great waste of computing resources. Second, random initialization cannot distinguish the inherent characteristics of different modal images. Infrared images are mainly based on thermal radiation information, and their frequency energy is concentrated in the low-frequency region; while visible light images contain rich texture details, and their high-frequency components are more significant. Using the same random initialization strategy to process these two image modalities with completely different physical characteristics undoubtedly increases the learning burden of the network.

[0004] More deeply, the existing initialization method lacks physical interpretability. The first layer of convolution kernel of the network directly processes the original pixels and should bear the responsibility of extracting basic features, but the kernel weight of random initialization looks like meaningless noise patterns. This not only affects the upper limit of the model performance, but also brings difficulties to the understanding and debugging of the model.

[0005] From the perspective of frequency domain analysis, the frequency components of an image provide a more essential feature description than the spatial domain. As a newly emerging signal processing technique, SAFD can adaptively select the optimal basis function to express the inherent frequency characteristics of the signal, which provides a powerful mathematical tool for extracting essential features from images. However, the existing technology has not established an effective conversion channel from the SAFD frequency domain features to the deep learning network weights.

[0006] Therefore, a new initialization paradigm is needed to fully utilize the prior knowledge of multi-modal images and provide a physically meaningful optimization starting point for deep fusion networks. The present invention is proposed in such a technical background, aiming to completely change the initialization method of deep networks through frequency domain feature analysis and conversion. SUMMARY

[0007] The purpose of the present invention is to overcome the fundamental defect of the existing deep neural network initialization method, which lacks physical meaning and prior knowledge, and to provide an image fusion network kernel initialization method based on frequency domain feature extraction. The core innovation of this method is to establish a direct mapping channel from image frequency domain features to network kernel weights, extract essential frequency domain features from source images through adaptive Fourier decomposition technology, and convert them into convolution kernel weights with clear physical meaning.

[0008] The technical solution adopted by the present invention is a systematic frequency domain-spatial domain feature conversion method, whose theoretical basis is the perfect combination of frequency domain analysis theory in signal processing and weight optimization theory in deep learning. This method realizes the complete conversion process from the original image to the optimized weight through five carefully designed steps, each step contains deep theoretical insights and engineering optimization:

[0009] Step 1: Image preprocessing and blocking. This step uses advanced image registration algorithms and adaptive contrast enhancement techniques to ensure that the input infrared and visible light images are accurately aligned in space and radiation characteristics. First, the image is standardized to linearly map its pixel values to the [0, 1] interval, eliminating radiation differences caused by different imaging devices. Then, a content-aware image blocking strategy is used to dynamically adjust the block size according to the local complexity of the image - smaller blocks are used in texture-rich areas to preserve details, and larger blocks are used in smooth areas to improve processing efficiency. This adaptive blocking method ensures the integrity of feature extraction while optimizing the allocation of computing resources.

[0010] Step 2: SAFD processing. A fast Fourier transform (FFT) based SAFD algorithm is applied to each image patch for frequency domain decomposition. The core advantage of SAFD is that it can adaptively select the most suitable basis functions to describe the current signal characteristics, rather than relying on a fixed set of basis functions. By constructing an over-complete atom dictionary and using the maximum inner product criterion, the algorithm can extract the key frequency components in the image patch layer by layer. The low-frequency components capture the macro structure and contour information of the image, while the high-frequency components retain the detailed texture and edge features. Finally, a set of complex-valued decomposition coefficients is output, including the atom sequence an and the corresponding coefficient matrix coef.

[0011] Step 3: Coefficient post-processing and feature aggregation. After SAFD decomposition, each image patch obtains a set of complex-valued decomposition coefficients, which completely describe the frequency domain characteristics of the image patch. This step first extracts the real part of the coefficients and converts it to the real number domain for subsequent processing. Then, an attention-weighted feature aggregation mechanism is used to intelligently fuse the frequency domain features of all image patches. Instead of simple arithmetic averaging, a weighted average is performed based on the information content and importance of each image patch, ensuring that the final global feature vector represents the most essential and critical frequency domain characteristics of the entire image.

[0012] Step 4: Convolution kernel generation and spatial domain conversion. This is the core technical link of the invention, which realizes the clever conversion from frequency domain features to spatial domain convolution kernels. The global feature vector coef_avg obtained in step 3 is input into a dedicated convolution kernel extraction function. The core operation of this function is to reshape the one-dimensional feature vector directly into a four-dimensional tensor with dimensions (width, height, input channel number, output channel number), thereby generating a convolution kernel weight matrix that can be directly used for convolution neural network calculations.

[0013] Step 5: Network weight initialization and propagation. The generated specialized convolution kernels are accurately assigned to the bottom convolution layers of the corresponding branches of the fusion network. The convolution kernels generated for infrared image features are specifically used to initialize the infrared branch, and the convolution kernels generated for visible light image features are specifically used to initialize the visible light branch. This modal-specific initialization strategy ensures that the network has optimal feature extraction capabilities from the beginning, laying a solid foundation for subsequent deep feature fusion. The main technical features of the invention are as follows:

[0014] Further, the convolution kernel generation in step 4 is achieved through tensor reshaping, and its operation can be represented as:

[0015] K = reshape(v avg ,(k,k,C in ,C out ))

[0016] where k is the size of the convolution kernel, Cin and C out are the input and output channel numbers, respectively.

[0017] Further, the converted infrared branch convolution kernel in step 4 is a 3x3 real matrix, and the specific values are as follows:

[0018] [83.6483, 0.4412, 0.0309,

[0019] 9.5124, 0.0777, -0.0048,

[0020] -0.7233, -0.0483, -0.0191]

[0021] The visible light branch convolution kernel is also a 3x3 real matrix, and the specific values are as follows:

[0022] [127.9718, -0.2880, -0.0028,

[0023] -5.8389, 0.0067, -0.0080,

[0024] 0.1897, 0.0564, 0.0200]

[0025] These values are statistical results obtained after the frequency domain decomposition and conversion process on a large number of image samples, which can effectively capture the typical features of each modal image.

[0026] It should be particularly noted that the above-provided infrared and visible light branch convolution kernel values are specific examples generated based on example image data through the SAFD process, and are used to show the form of the output results of the method. In actual application, the convolution kernel matrix values generated through the SAFD frequency domain decomposition and conversion process will change accordingly according to the different input image contents, in order to better capture the frequency domain features of specific images. The advantage of the method is that it can adaptively generate special convolution kernels with physical meaning according to different input image data, rather than using fixed general values.

[0027] Further, the SAFD process in step 2 is implemented using a SAFD algorithm based on fast Fourier transform, which uses fast Fourier transform to speed up the inner product calculation between dictionary atoms and signals. The atom selection mechanism is based on the maximum inner product criterion under the greedy algorithm framework, which ensures that the most energetic frequency component is extracted at each decomposition, thereby achieving efficient sparse representation of image frequency domain features.

[0028] Further, the image fusion network served by the method is a dual-branch encoder-single decoder network structure designed using a reparameterization technique; the convolutional layer initialized using the method is located at the front end of the encoder branch and is the first layer of operation for the network to process the original input image, and the quality of the weight initialization directly affects the extraction quality of subsequent features and the convergence speed of the entire training process. These convolution kernels actually play the role of a feature extraction preprocessor.

[0029] Further, the image fusion network uses a multi-objective loss function combining multiple visual fidelity indicators for end-to-end optimization; the loss function is a linear weighted sum of structural similarity loss (SSIM), pixel-level L1 norm reconstruction loss, edge consistency loss based on image gradients, and region weighted mask loss generated using saliency detection.

[0030] The advantages of the method include:

[0031] (1) clear physical meaning and strong interpretability: the invention completely abandons the black-box random initialization method and constructs the convolution kernel from the physical properties of the image data itself, making the first layer of operation of the network have a clear physical meaning - that is, it is specifically used to extract frequency domain features of a particular modality. This interpretability not only helps to understand the working principle of the model, but also provides a clear direction for subsequent model optimization.

[0032] (2) accelerate network convergence and improve training efficiency: the network is provided with an initial point close to the optimal solution, significantly reducing the number of iterations and computational resources required for training, speeding up model deployment, and being particularly suitable for end-side applications with high real-time requirements.

[0033] (3) improve fusion performance: data-driven convolution kernels can more effectively extract the inherent features of infrared and visible light images, providing higher quality feature maps for subsequent fusion layers, resulting in better fusion results in both objective indicators and subjective visual perception.

[0034] (4) generalization and flexibility: the invention provides a general initialization strategy that does not depend on a specific network architecture and can be widely applied to various image fusion models based on convolutional neural networks. By simply replacing the initialization method, performance can be improved without any additional computational overhead. BRIEF DESCRIPTION OF DRAWINGS

[0035] The features and advantages of the present invention will be more clearly understood through reference to the following drawings, which are illustrative and not intended to be limiting in any way, in which

[0036] Figure 1A flow chart of an image fusion network convolution kernel initialization method based on frequency domain feature extraction is provided for the embodiments of the present application.

[0037] Figure 2 A SAFD generation process schematic diagram is provided for the embodiments of the present application.

[0038] Figure 3 A convolution kernel generation and network initialization schematic diagram is provided for the embodiments of the present application.

[0039] Figure 4 A visual effect diagram of the final fusion image obtained using different initialization methods is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0041] Figure 1 The overall flow chart of the present application is shown. As shown in the figure, the present application provides an image fusion network convolution kernel initialization method based on frequency domain feature extraction, which includes the following five core steps:

[0042] Step 1: Image preprocessing and blocking.

[0043] The present application uses registered infrared and visible light image pairs in a public dataset (such as M3FD) as input. First, the input images are subjected to gray scale standardization, and their pixel values are normalized to the [0, 1] interval, as follows:

[0044]

[0045] where I is the original image, I min and I max are the minimum and maximum pixel values of the image, respectively. Subsequently, each image is divided into multiple non-overlapping or overlapping image blocks (such as 16x16 pixels), and each image block is regarded as an independent signal processing unit for subsequent frequency domain feature analysis.

[0046] Step 2: SAFD processing.

[0047] For each image patch, SAFD algorithm based on fast Fourier transform is adopted for decomposition. The algorithm accelerates the calculation of inner product of signal and each atom in the redundant dictionary through fast Fourier transform, and selects the optimal atom layer by layer according to the maximum inner product criterion (greedy algorithm), and outputs the decomposition coefficient coef and the atom sequence an. This step converts the image patch from the spatial domain to the frequency domain, and extracts its multi-level feature representation.

[0048] Step 3: coefficient post-processing and feature aggregation.

[0049] For the obtained complex coefficient matrix coef, the real part is extracted to obtain coef_real in view of the relatively minor role of complex phase information in image texture expression. Then, the real part coefficients of all image patches are averaged in the first dimension to obtain a global feature vector coef_avg:

[0050]

[0051] The vector converges the statistical commonness of the frequency domain features of the source image in different local areas, and is the most essential and representative frequency domain feature summary. Convolution kernel generation and spatial domain conversion. This is the core technical link of the application, which realizes the ingenious conversion from frequency domain features to spatial domain convolution kernels.

[0052] Step 4: convolution kernel generation and spatial domain conversion.

[0053] The global feature vector coef_avg obtained in step 3 is input into a dedicated convolution kernel extraction function. The core operation of the function is to reshape the one-dimensional feature vector directly into a four-dimensional tensor with dimensions (width, height, input channel number, output channel number), thereby generating a convolution kernel weight matrix that can be directly used for convolution neural network calculation. Thus, the conversion from frequency domain features to spatial domain convolution kernel weights is completed.

[0054] Step 5: network weight initialization.

[0055] The infrared convolution kernel weight matrix generated in step 4 is assigned to the weight of the first convolution layer of the network infrared branch by direct assignment method; similarly, the visible light convolution kernel is assigned to the corresponding convolution layer of the visible light branch. The values are as follows:

[0056] Infrared branch convolution kernel:

[0057] [83.6483, 0.4412, 0.0309,

[0058] 9.5124, 0.0777, -0.0048,

[0059] -0.7233, -0.0483, -0.0191]

[0060] Visible light branched convolution kernel:

[0061] [127.9718, -0.2880, -0.0028,

[0062] -5.8389, 0.0067, -0.0080,

[0063] [0.1897, 0.0564, 0.0200]

[0064] It should be noted that the infrared and visible light branch convolution kernel values ​​provided above are specific examples generated through the SAFD process based on example image data, used to demonstrate the output format of this method.

[0065] At this point, the initialization of the lowest-level convolutional kernels of the network is complete. These convolutional kernels are now capable of extracting specific frequency domain features of a modality, and the network can proceed with normal training based on this.

[0066] Figure 2 A detailed schematic diagram of the SAFD process provided in an embodiment of the present invention. For example... Figure 2 As shown, for each image patch X, the Fast Adaptive Fourier Decomposition algorithm is used for decomposition. This algorithm accelerates the calculation of the inner product of the signal with each atom in a pre-constructed dictionary of redundant atoms D through Fast Fourier Transform. <X,e a Its atom selection mechanism is based on the maximum inner product criterion within a greedy algorithm framework, that is, in each level of decomposition, atoms that satisfy the following formula are selected:

[0067]

[0068] in, It is the signal after the k-th level decomposition, e a It is the parameter a The process is iterative, peeling away the most energetic frequency components from the image patch layer by layer, ultimately outputting two sets of key parameters: the atomic sequence {a1, a2, ..., a...}. N} and the corresponding decomposition coefficient matrix C∈C P×N (P is the number of blocks, N is the number of decomposition layers), thus completing the multi-scale feature representation from the spatial domain to the frequency domain.

[0069] Figure 3 This diagram illustrates the convolutional kernel generation and network initialization process provided in an embodiment of the invention. The global feature vector `coef_avg` is input into the convolutional kernel extraction function. The core operation of this function is reshaping, that is, reshaping the one-dimensional vector into a dimension (k, k, C). in C out A four-dimensional tensor filter that can be directly used as a convolution kernel:

[0070] filter = reshape(coef_avg),

[0071] where k is the size of the convolution kernel (height and width), C in is the number of input channels, C out is the number of output channels.

[0072] As shown in Figure 4 , the fusion results of the M3FD dataset are shown. It can be seen visually that the output fusion image of the network initialized and trained by the method of the present application has significant advantages in many aspects: 1) detail preservation: better preservation of texture details (such as ground texture, vegetation) in the visible light image; 2) feature enhancement: more prominent display of thermal target information (such as vehicles, pedestrians) in the infrared image; 3) naturalness: overall contrast balance, complete structure information, and more natural visual perception.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not limiting; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for initializing convolutional kernels in an image fusion network based on frequency domain feature extraction, characterized in that, This method constructs convolution kernels with explicit physical meaning by analyzing the essential frequency domain characteristics of images, specifically including the following steps: Step 1: Image preprocessing and segmentation. The registered infrared and visible light images are subjected to grayscale normalization and contrast normalization respectively to eliminate intermodal differences. Then, each image is evenly divided into multiple image blocks of the same size, and each image block is regarded as an independent processing unit. Step 2: SAFD processing. The SAFD algorithm based on Fast Fourier Transform is used to decompose each image block in the frequency domain. This process extracts low-frequency contour information and high-frequency detail features from the image block layer by layer by constructing a redundant atom dictionary and using the maximum inner product criterion for atom selection. Finally, a set of complex decomposition coefficients is output, including the atom sequence an and the corresponding decomposition coefficient matrix coef. Step 3: Coefficient post-processing and feature aggregation. The real part of the complex coefficient matrix coef obtained above is extracted to transform it from the complex domain to the real domain, resulting in the real coefficient matrix coef_real. Subsequently, the arithmetic mean of the matrix is ​​calculated along the image patch dimension to obtain a compressed vector coef_avg that can represent the global frequency domain features of the entire image. Step 4: Convolution kernel generation and spatial domain transformation. The global feature vector coef_avg obtained in Step 3 is input into a dedicated convolution kernel extraction function. This function first reshapes the one-dimensional feature vector into a two-dimensional structure, then verifies its effectiveness in the spatial domain through convolution operations, and performs geometric transformation processing to finally generate a fixed-size weight matrix that can be directly applied to the convolutional neural network, i.e., the convolution kernel. Step 5: Network weight initialization. The convolutional kernel weight matrix generated in Step 4 is directly used to initialize the convolutional layer weights of the corresponding branches in the image fusion neural network encoder. Among them, the convolutional kernel generated from infrared image data is specifically used to initialize the convolutional layer of the infrared branch, and the convolutional kernel generated from visible light image data is specifically used to initialize the convolutional layer of the visible light branch, thereby providing the network with an initial point with good prior knowledge.

2. The method according to claim 1, characterized in that, The infrared branch convolution kernel generated in step 4 is a 3x3 real matrix, with the following specific values: [83.6483,0.4412,0.0309, 9.5124,0.0777,-0.0048, -0.7233,-0.0483,-0.0191] The visible light branch convolution kernel is also a 3x3 real number matrix, with the following specific values: [127.9718,-0.2880,-0.0028, -5.8389,0.0067,-0.0080, 0.1897,0.0564,0.0200] These values ​​are statistical results obtained by performing the frequency domain decomposition and transformation process on a large number of image samples, and can effectively capture the typical features of each modal image. It should be noted that the infrared and visible light branch convolution kernel values ​​provided above are specific examples generated through the SAFD process based on sample image data, used to demonstrate the output format of this method. In practical applications, the values ​​of the convolution kernel matrix adaptively generated through the SAFD frequency domain decomposition and transformation process will vary according to the content of the input image, in order to better capture the frequency domain features of a specific image. The advantage of this method lies in its ability to adaptively generate physically meaningful dedicated convolution kernels based on different input image data, rather than using fixed universal values.

3. The method according to claim 1, characterized in that, The SAFD process in step 2 is specifically implemented using the SAFD algorithm based on Fast Fourier Transform. This algorithm uses Fast Fourier Transform to accelerate the inner product calculation between dictionary atoms and signals. Its atom selection mechanism is based on the maximum inner product criterion under the greedy algorithm framework, ensuring that each decomposition can extract the most energetic frequency component.

4. The method according to claim 1, characterized in that, The core operation of the convolution kernel extraction function in step 4 is to directly reshape the feature vector into a four-dimensional tensor with dimensions of (width, height, number of input channels, and number of output channels), thereby generating a convolution kernel weight matrix that can be directly used for convolutional neural network calculations.

5. The method according to claim 1, characterized in that, The image fusion network served by the method is a dual-branch encoder-single decoder network structure designed with reparameterization technology. The convolutional layer initialized using the method is located at the front of the encoder branch and is the first layer operation of the network in processing the original input image. The quality of its weight initialization directly affects the quality of subsequent feature extraction and the convergence speed of the entire training process.

6. The method according to claim 1 or 5, characterized in that, The image fusion network employs a multi-objective loss function that combines multiple visual fidelity metrics for end-to-end optimization. This loss function is a linear weighted sum of structural similarity loss (SSIM), pixel-level L1 norm reconstruction loss, edge consistency loss based on image gradient, and region-weighted mask loss generated using saliency detection. The network parameters are jointly optimized through backpropagation to ensure that the fused image achieves optimal performance across multiple dimensions, including structure, pixels, edges, and visual attention regions.

7. The method according to claim 1, characterized in that, The overall process of the method is a combination of data-driven and model-driven approaches. Instead of using random numbers or conventional mathematical initialization methods, it learns and extracts the most representative frequency domain features from actual training image data and converts them into spatial domain convolution kernels, thereby providing a high-performance and highly interpretable initialization strategy for neural networks.

Images