An image fusion method and system based on three-branch feature reconstruction and fusion

Abstract

The present application relates to the technical field of image processing, and more particularly to an image fusion method and system based on three-branch feature reconstruction and fusion, which first maps a first source image and a second source image to a high-dimensional feature space, extracts common feature maps representing scene structure using a spatial response intersection; then calculates redundant components of initial features in the direction of common features in a private branch, and further obtains first source and second source private feature maps strictly orthogonal to the common features; finally generates a fused image through channel splicing and decoding; realizes deep decoupling of features, completely eliminates feature entanglement and interference between modalities; while completely preserving the thermal targets and texture details of the source images, effectively eliminates ghosting and halo artifacts at the edges of the fused image, greatly improves the sharpness of the target outline and the contrast and clarity of the overall image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image fusion method and system based on three-branch feature reconstruction and fusion. Background Technology

[0002] In existing feature-decoupling-based fusion schemes, a three-branch network architecture is typically used to extract the first source private features, visible light private features, and the common features of the two respectively. This type of method separates information of different attributes by constructing independent feature extraction channels so as to retain the unique advantages of each modality in the fusion stage, while enhancing the overall structural expression of the scene. In this way, it is expected to obtain a high-quality fused image with both clear target outlines and rich background details.

[0003] However, although the existing three-branch decoupling method has improved the fusion effect to some extent, there are still obvious problems in practical applications. Specifically, the existing feature extraction and separation mechanism often fails to achieve true feature orthogonality, resulting in inconsistent extracted common features. When dealing with complex scenarios, private information unique to a certain modality is easily mixed into the common feature branches, causing ambiguity and overlap in feature definitions.

[0004] Incomplete feature separation directly affects the final fusion quality. When shared features carrying redundant private information are combined with the original private features for image reconstruction, the repeated superposition of information will cause difficult-to-eliminate ghosting or halo phenomena at the edges of objects in the fused image, which seriously affects the contrast and clarity of the image, and thus interferes with the accuracy of subsequent target detection or recognition tasks. Summary of the Invention

[0005] The main objective of this invention is to provide an image fusion method based on three-branch feature reconstruction and fusion, which aims to solve the technical problem of incomplete feature separation in existing image fusion methods.

[0006] To achieve the above objectives, this invention provides an image fusion method based on three-branch feature reconstruction and fusion, the method comprising the following steps: A first source image and a second source image to be fused are obtained, and the first source image and the second source image are respectively mapped to a high-dimensional feature space to obtain an initial feature map, wherein the initial feature map includes a first source initial feature map and a second source initial feature map; Based on the first source initial feature map and the second source initial feature map, a common feature backbone branch is constructed, and the intersection of the first source initial feature map and the second source initial feature map in spatial response is obtained to extract the common feature map representing the scene structure and background contour. Private branches are constructed based on the common feature map. In the private branches, redundant components of the initial feature map in the direction of the common feature map are obtained. Based on the redundant components, a first source private feature map and a second source private feature map orthogonal to the common feature map are obtained. The private branches include a first source private branch and a second source private branch. According to a preset feature recombination strategy, the first source private feature map, the second source private feature map, and the common feature map are channel-stitched and fused to obtain a target fused image containing the first source target information and the second source texture details.

[0007] To achieve the above objectives, the present invention also provides an image fusion system based on three-branch feature reconstruction and fusion, the system comprising: An image acquisition module is used to acquire a first source image and a second source image to be fused, and to map the first source image and the second source image to a high-dimensional feature space respectively to obtain an initial feature map, wherein the initial feature map includes a first source initial feature map and a second source initial feature map; A common feature extraction module is used to construct a common feature backbone branch based on the first source initial feature map and the second source initial feature map, and to obtain the intersection of the first source initial feature map and the second source initial feature map in spatial response, so as to extract a common feature map that represents the scene structure and background contour. A private feature extraction module is used to construct private branches, a first source residual private branch and a second source residual private branch according to the common feature map. In the private branches, the redundant components of the initial feature map in the direction of the common feature map are obtained, and the first source private feature map and the second source private feature map orthogonal to the common feature map are obtained according to the redundant components. The private branches include the first source private branch and the second source private branch. The fusion decoding module is used to perform channel splicing and fusion decoding on the first source private feature map, the second source private feature map and the common feature map according to a preset feature recombination strategy, so as to obtain a target fusion image containing the first source target information and the second source texture details.

[0008] The beneficial effects that this invention can achieve are as follows: This invention maps the first and second source images to be fused onto a high-dimensional feature space to obtain an initial feature map. It then extracts a common feature map representing the scene structure and background contour using the intersection of their spatial responses. Furthermore, it calculates the redundant components of the initial feature map along the direction of the common feature map in the private branches, thereby obtaining a first-source private feature map and a second-source private feature map that are strictly orthogonal to the common feature map in mathematical properties. Finally, it generates the target fused image through channel stitching and fusion decoding. This effectively solves the technical problems of feature entanglement and incomplete separation commonly found in existing image fusion techniques, particularly addressing the edge artifacts and blurred details in the fused image caused by residual background common information in the private features. It achieves deep decoupling and precise recombination at the feature level. By acquiring the intersection of spatial responses, it is ensured that the shared feature map contains only robust structural information shared by the two modalities. Furthermore, by calculating redundant components and acquiring orthogonal private feature maps, background interference mixed in the private branches is forcibly removed from the mathematical level. This allows the first source private feature map to focus on expressing independent target information, while the second source private feature map focuses on expressing independent texture details. This orthogonal feature representation method avoids visual noise caused by repeated information superposition during the final feature recombination process. It ensures that the generated fused image can retain the significant target intensity of the first source while losslessly presenting the fine texture of the second source, significantly improving the edge sharpness, contrast, and overall visual clarity of the fused image. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating the method in Embodiment 1 of the present invention; Figure 2 This is a structural block diagram of the system in Embodiment 8 of the present invention.

[0010] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0012] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0013] In this invention, unless otherwise explicitly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0014] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the meaning of "and / or" throughout the text includes three parallel solutions; for example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied simultaneously. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention. Example

[0015] As attached Figure 1 As shown, this embodiment provides an image fusion method based on three-branch feature reconstruction and fusion, the method including the following steps: A first source image and a second source image to be fused are obtained, and the first source image and the second source image are respectively mapped to a high-dimensional feature space to obtain an initial feature map, wherein the initial feature map includes a first source initial feature map and a second source initial feature map; Based on the first source initial feature map and the second source initial feature map, a common feature backbone branch is constructed, and the intersection of the first source initial feature map and the second source initial feature map in spatial response is obtained to extract the common feature map representing the scene structure and background contour. Private branches are constructed based on the common feature map. In the private branches, redundant components of the initial feature map in the direction of the common feature map are obtained. Based on the redundant components, a first source private feature map and a second source private feature map orthogonal to the common feature map are obtained. The private branches include a first source private branch and a second source private branch. According to a preset feature recombination strategy, the first source private feature map, the second source private feature map, and the common feature map are channel-stitched and fused to obtain a target fused image containing the first source target information and the second source texture details.

[0016] In existing feature-decoupling-based fusion schemes, feature extraction and separation mechanisms struggle to achieve true feature orthogonality, resulting in inconsistent extracted common features. When dealing with complex scenes, private information specific to a particular modality may be mixed into the common feature branches, causing ambiguity and overlap in feature definitions. This incomplete feature separation directly affects the final fusion quality. When common features carrying redundant private information are combined with private features for image reconstruction, the repeated superposition of information causes ghosting or halo phenomena at object edges in the fused image, affecting image contrast and clarity, and thus interfering with the accuracy of subsequent target detection or recognition tasks.

[0017] Based on the above problems, this embodiment provides an image fusion method based on three-branch feature reconstruction and fusion. The method constructs a decoupled network architecture that includes a common feature backbone branch and two private feature branches. Its core lies in using the mathematical mechanism of vector orthogonal projection to force the information in the multimodal source image to be separated into statistically independent modal private features and spatially consistent common features in the feature space, thereby solving the problems of edge artifacts and contrast reduction caused by feature entanglement in traditional fusion methods.

[0018] Understandably, in practical engineering deployment scenarios, such as all-weather security monitoring or vehicle-mounted driver assistance systems, the first source image is usually selected from image data collected by an infrared thermal imaging sensor. This type of image is based on the thermal radiation difference imaging of an object, which can effectively overcome the interference of insufficient light or smoke obstruction and accurately capture targets with heat source characteristics. The second source image is usually selected from image data collected by a visible light sensor. This type of image is based on the reflected light from the surface of an object and contains rich color, texture and detail information.

[0019] It is also understandable that the original image data only reflects the numerical distribution of light intensity or thermal radiation intensity and lacks semantic understanding of the image content. In order to overcome the defects of directly fusing in the pixel gray space, which easily introduces noise and makes it difficult to align semantic information, this embodiment maps the two-dimensional image matrix to a high-dimensional feature manifold through a feature extraction network.

[0020] Specifically, the feature mapping process uses a convolutional neural network for nonlinear transformation, and its mathematical expression logic is as follows: ; In the formula, This represents the initial feature map of the generated k-th source, with the data source being a three-dimensional feature tensor containing multiple channels output by the feature extraction network; k represents the mode index. When the value is 1, it represents the first source (such as infrared mode), and when the value is 2, it represents the second source (such as visible light mode).

[0021] The input k-th original source image is represented by the image matrix acquired by the front-end sensor and preprocessed (such as registration and denoising). This represents the convolutional kernel weight matrix of the convolutional layer. The data source is the network parameters optimized during the training phase through the backpropagation algorithm, which is used to extract primary visual features such as edges and corners. * represents the convolution operator, which physically means performing a weighted summation on a local region of an image through a sliding window; This represents the bias term parameter, which is a constant term obtained from network training and is used to adjust the trigger threshold of the activation function. This indicates a batch normalization operation, used to unify feature distributions to accelerate network convergence. This represents a non-linear activation function (e.g., Leaky ReLU), used to introduce non-linear factors to enhance the network's ability to fit complex features.

[0022] Based on the above expression, the low-level pixel information is transformed into high-level semantic features. Through the combination of convolution operation and non-linear activation, the network can extract feature representations with semantic attributes (such as gradient direction, texture mode, etc.) from simple gray values. This processing logic provides a data foundation with rich semantic information for subsequent feature decoupling, enabling the algorithm to separate based on the content attributes of features rather than simple pixel values, which significantly improves the robustness of the algorithm in complex lighting change scenarios.

[0023] Subsequently, in the multi-source image fusion task, common features specifically refer to the scene geometric structures that objectively exist in all source images, such as the physical outlines of buildings, the boundary lines of roads, and the horizon. These information should exhibit significant response values ​​in both the infrared feature space and the visible light feature space. In order to accurately extract this part of the intersection information from the mixed features, this embodiment does not use the traditional shared weight encoder, but designs a gating attention mechanism based on spatial response interaction.

[0024] This mechanism determines common features by calculating the product of the responses of two modal features at spatial locations. The specific calculation formula is as follows: ; In the formula, This represents the final extracted common feature map, with data sourced from calculation results, which characterizes the general skeleton structure of the scene; This represents the first source initial feature map, which originates from the output of the previous feature mapping step.

[0025] This represents the second source initial feature map, whose data source is the output of the previous feature mapping step; This represents the average pooling operation along the channel dimension. The data source is a standard pooling algorithm, used to compress multi-channel feature tensors into a single-channel spatial saliency map.

[0026] This represents the Hadamard product; that is, the element-wise multiplication of matrices, used to calculate the intersection of spatial responses. This represents the Sigmoid activation function, whose data source is a preset mathematical function. It is used to map the product result to probability weights between 0 and 1 to generate a spatial attention mask. This indicates a feature fusion convolutional layer, whose data source is a convolutional operation containing learnable parameters, used to initially integrate the concatenated features; This indicates a concatenation operation along the channel dimension; it is used to physically combine two features in a dimensional manner. × represents the broadcast multiplication operation between the scalar weights and the feature tensor; it is used to apply an attention mask to the feature map.

[0027] It should be noted that in the spatial saliency map, the product operation will only produce a large value when both the first and second sources have high response values ​​at the same coordinate position (i.e., both "see" the object), thereby activating the attention weight at that position. Conversely, if a feature appears only in a single modality (such as a thermal target unique to infrared or a surface pattern unique to visible light), its response in another modality will approach zero, causing the product result to be suppressed. This mathematically forces the filtering of modality-specific noise interference, ensuring that the extracted common feature map contains only pure scene structure information. This logic has excellent adaptability in special scenarios such as complex nighttime environments, effectively eliminating false contours caused by infrared overexposure areas and providing a high-precision benchmark reference for subsequent feature decoupling.

[0028] Furthermore, in order to completely eliminate the edge artifact problem caused by incomplete feature separation in traditional methods, this embodiment introduces the Schmitt orthogonalization principle in linear algebra, treating the feature map as a vector in a high-dimensional vector space. By calculating the projection and performing a subtraction operation, the private features are forced to be perpendicular to the common features in mathematical properties, thereby ensuring that the information contained in the two do not overlap.

[0029] Specifically, this step first calculates the projection components (i.e., redundant components) of the initial feature map along the common feature map direction, and then removes them. The calculation logic is as follows: ; ; In the formula, It represents the redundant component of the initial feature map of the k-th path in the direction of the common feature map. The data source is the calculation result, and the physical meaning is the background information mixed in the initial feature. This represents the tensor inner product operation; the data source is mathematical calculation, used to calculate the correlation strength between two feature maps.

[0030] It represents the square of the L2 norm; the data source is mathematical calculation, that is, calculating the energy modulus of the common feature map.

[0031] This represents the numerical stability constant (usually taken as 1e-8); the data source is a preset hyperparameter, used to prevent computational overflow caused by a zero denominator.

[0032] • Represents the scalar multiplication operation between scalar coefficients and tensors.

[0033] This represents the final obtained k-th private feature map; the data source is the calculation result, i.e., the orthogonalized modal-specific features.

[0034] Based on the above expression, a scalar projection coefficient is calculated by the ratio of the inner product to the square of the modulus. This coefficient physically quantifies "what proportion of information in the initial features belongs to the common background." Secondly, by multiplying this coefficient by the common feature reference vector, the background contour components (i.e., redundant components) contained in the initial features are accurately reconstructed. Finally, through subtraction, these redundant components are completely removed from the initial features. The effect of this logic is that it mathematically guarantees the orthogonality (i.e., zero correlation) between private and common features. This means that in subsequent image reconstruction, the object's edge contour (provided by common features) and the object's internal properties (provided by private features) will be independently controlled by different channels, without interference. This solves the halo or ghosting phenomenon commonly seen at object edges in traditional fused images, significantly improving image sharpness and target recognizability, making it particularly suitable for autonomous driving obstacle detection scenarios with extremely high edge accuracy requirements.

[0035] Furthermore, after completing the fine decoupling of features, the system obtains three sets of unrelated high-quality features: a common feature map representing the geometric skeleton of the scene, a first-source private feature map representing pure thermal radiation intensity, and a second-source private feature map representing pure surface texture details. At this point, feature reorganization no longer requires a complex conflict coordination mechanism, because the redundancy between features has been eliminated in the previous step.

[0036] The logical formula for feature recombination and decoding is as follows: ; In the formula, This represents the final generated target fused image; the data source is the output of the decoding network. The decoder network represents the data source, which is a constructed deep neural network module, typically consisting of multiple layers of deconvolution or upsampling modules, used to map features back to pixel space. This represents a tensor concatenation operation along the channel dimension; the data source is a data processing operation used to stack features from different branches in the depth direction. This represents the first source private feature map; the data source is the output of the orthogonal stripping step.

[0037] This represents the second source private feature map; the data source is the output of the orthogonal stripping step. This represents the shared feature map; the data source is the output of the intersection extraction step.

[0038] Based on the above expression, channel splicing can preserve the complete information of each branch to the greatest extent. The task of combining this information is entrusted to the decoder network to learn through backpropagation. The decoder can use its powerful nonlinear mapping capability to gradually restore the abstract semantic features into visualized image pixels. Finally, the fused image output by the system retains both the high-contrast thermal target in the infrared image and the rich texture details in the visible light image. The overall structure is clear and there are no edge artifacts, realizing the complementary use of multimodal information.

[0039] Example 2: In this embodiment, obtaining the intersection of the first source initial feature map and the second source initial feature map in spatial response to extract a common feature map representing the scene structure and background contour includes: The first source initial feature map and the second source initial feature map are compressed in the channel dimension to obtain the first source space descriptor and the second source space descriptor. Element-wise multiplication is performed on the first source spatial descriptor and the second source spatial descriptor to enhance the common response region and suppress the unilateral response region, and to obtain the spatial attention mask. The linear combination of the first source initial feature map and the second source initial feature map is weighted and filtered according to the spatial attention mask to obtain the common feature map.

[0040] Understandably, since the initial feature map is a high-dimensional tensor containing dozens or even hundreds of channels, directly performing interactive computation on all channels would be too computationally intensive and easily introduce noise. Therefore, the system first compresses the first and second source initial feature maps along the channel dimension using average pooling, flattening the three-dimensional feature tensor into two-dimensional first and second source spatial descriptors. These two descriptors no longer contain specific semantic content, but purely reflect the energy distribution of features in the spatial plane, i.e., they are used to characterize "which locations in the image contain significant information".

[0041] Subsequently, the system performs element-wise multiplication to generate a spatial attention mask. The first source spatial descriptor and the second source spatial descriptor are multiplied at their corresponding pixel locations. Mathematically, if a pixel location has a high response only in the first source (e.g., an infrared heat source) and a very low response in the second source (e.g., a dark background), the product will approach zero. Conversely, the product will only produce a large value when both sources have high responses at that location (e.g., a common object contour). In this way, the system effectively suppresses unilateral response regions (i.e., modal private regions) and enhances shared response regions. The product result is then mapped to the zero-to-one interval by a nonlinear activation function to form the final spatial attention mask.

[0042] Finally, the spatial attention mask is used to perform weighted filtering on the linear combination result of the initial features. Specifically, the first source initial feature map and the second source initial feature map are concatenated or summed in the channel dimension to form a hybrid feature base containing all potential information. Then, the spatial attention mask generated above is used to perform point-by-point weighting on the base. Regions with values ​​close to 1 (common regions) in the mask are retained, while regions with values ​​close to 0 (private regions) are suppressed, thereby outputting a pure common feature map.

[0043] Understandably, in the backpropagation process of deep learning, the gradient of the multiplication node is related to another operand. The gradient update of the first source feature will be constrained by the strength of the second source feature. If a feature is only significant in the first source (e.g., a thermal target unique to infrared), the corresponding second source feature value is zero. Then, when calculating the common feature, the gradient of that region will also approach zero. This means that the network cannot understand those features that belong only to one-sided modalities during training, thus forcing the common feature branch to only focus on the structural information shared by the two modalities.

[0044] This logic exhibits significant adaptability to specific scenarios corresponding to the technical solutions. For example, in nighttime surveillance scenarios, visible light images are often completely black (with most areas having a response of 0), while infrared images show clear targets. If traditional additive fusion is used, noise from visible light is introduced into the shared features. However, using the multiplicative intersection logic of this formula, since the visible light response is 0, the product result is automatically 0, thus naturally filtering out low-quality noise from the visible light channel. This ensures that the extracted shared features (such as the horizon outline) are always clear and robust. This mutual gating mechanism allows the shared feature map to serve as a reference for subsequent orthogonal projection, greatly reducing error propagation during feature decoupling.

[0045] Example 3: In this embodiment, obtaining the redundant components of the initial feature map in the direction of the common feature map in the private branch, and obtaining the first source private feature map and the second source private feature map orthogonal to the common feature map based on the redundant components, includes: The first source initial feature map is regarded as the target vector in the vector space, and the common feature map is regarded as the reference vector; Obtain the inner product between the target vector and the reference vector, and normalize the inner product according to the magnitude of the reference vector to obtain the first projection coefficient; Multiply the first projection coefficient by the reference vector to obtain the redundant component of the first source initial feature map in the direction of the common feature map; The redundant component is subtracted from the first source initial feature map to obtain the first source private feature map, such that the first source private feature map is mathematically perpendicular to the common feature map.

[0046] The steps for obtaining the second source private feature map are the same as those for obtaining the first source private feature map.

[0047] It should be noted that in the actual feature processing flow, after the system extracts the common feature map representing the scene structure and background contour through the preceding steps, the key task is to use this common feature map as a mathematical benchmark to purify the first source initial feature map (e.g., infrared features) and the second source initial feature map (e.g., visible light features). To achieve this, this embodiment introduces the orthogonal projection theory from functional analysis, treating the high-dimensional feature tensor as an entity in a vector space. By calculating the projection components and performing subtraction operations, the decoupling of the feature content is forcibly achieved.

[0048] It should also be noted that, for acquiring the first source private feature map, the system first treats the initial first source feature map as the target vector in the vector space, and the already extracted common feature map as the reference vector. In a mathematical and physical sense, the reference vector represents the background color shared by all modalities in the current scene, while the target vector is a mixture containing the background color and modality-specific information. In order to separate the unique information, the system first needs to quantize the projection of the target vector onto the reference vector, that is, to calculate the redundant components. This process is achieved by calculating the inner product between two feature tensors and normalizing it using the magnitude of the reference vector.

[0049] In the aforementioned corresponding expressions, the processing logic exhibits strong adaptability to specific scenarios corresponding to the technical solution. For example, in infrared and visible light fusion scenarios, infrared images often contain "human thermal targets" (private information) and "human contours" (shared information). Without orthogonal stripping, human contours will remain in the infrared private features. During fusion, the shared branch provides one contour, and the infrared private branch provides another. Even slight spatial misalignment between the two contours can cause ghosting at the edges of the fused image. However, using the orthogonal projection logic of this embodiment, the contours in the infrared private branch are forcibly cleared to zero, retaining only thermal information. This ensures that the edges of the final fused image are uniquely determined only by the shared feature branch, achieving a pixel-level sharp fusion effect.

[0050] Furthermore, in specific implementations, the system reuses the same orthogonal projection logic described above, using the second source initial feature map (such as visible light features) as the target vector and projecting it onto the same common feature reference vector.

[0051] The corresponding processing logic is as follows: calculate the inner product of the initial features of the second source and the common features and normalize it to obtain the second projection coefficient; use this coefficient to generate the redundant component of the second source (usually corresponding to the geometric structure information in the visible light image); finally subtract the redundant component from the initial features of the second source to obtain the private feature map of the second source (usually corresponding to pure color or surface texture details). This symmetrical design ensures that the private features of the two modalities are orthogonal to the same common feature reference, thereby constructing a rigorous orthogonal three-branch feature space, laying a solid mathematical foundation for subsequent high-quality complementary fusion.

[0052] Example 4: In this embodiment, the process of constructing private branches based on the shared feature map further includes: Obtain the cosine similarity between the first source private feature map and the shared feature map, and the cosine similarity between the second source private feature map and the shared feature map; Extract edge gradient maps from the first source image or the second source image, and obtain an edge weighted mask based on the edge gradient maps; The cosine similarity is spatially weighted according to the edge-weighted mask so that the orthogonality penalty weight of the image edge region is higher than that of the flat region.

[0053] Understandably, in the actual process of building a deep neural network model and training parameters, although the projection subtraction operation in the forward propagation process can achieve mathematically forced decoupling, in order to enable the encoder network to actively learn the orthogonal distribution characteristics between features and prevent nonlinear feature drift as the network layer deepens, this embodiment introduces an orthogonal regularization constraint mechanism based on edge perception. Its core is to identify high-frequency regions (i.e. edges) in the image that are prone to feature aliasing and to impose stronger independence penalties on these regions, thereby eliminating the root cause of artifact generation during the training phase.

[0054] In practical implementation, the system first needs to quantify the statistical correlation between private and shared features. Although decoupled features are obtained by subtracting the projection component in the preceding steps, the design of the loss function still requires explicit calculation of the cosine similarity between the first source private feature map and the shared feature map, as well as the cosine similarity between the second source private feature map and the shared feature map. Cosine similarity, as an indicator of the cosine value of the angle between two non-zero vectors in multidimensional space, can effectively characterize the degree of alignment of two feature tensors in direction. When the cosine similarity approaches zero, it indicates that the two feature vectors are perpendicular to each other in direction, that is, they have reached the ideal orthogonal and independent state; conversely, if the absolute value is large, it indicates that the private features still retain information similar to the shared features (such as background contours), or that private noise has been mixed into the shared features.

[0055] To further improve decoupling accuracy, this embodiment introduces an edge-weighted strategy, based on a key prior knowledge: in image fusion tasks, feature entanglement and artifacts tend to concentrate in high-frequency areas such as object contours and texture boundaries, while feature separation is relatively easy and less prone to visual defects in flat background areas. Therefore, the system extracts edge gradient maps from either the first or second source image. This typically involves convolving the source image using the Sobel, Prewitt, or Laplacian operators to calculate the spatial rate of change of pixel grayscale values, thereby obtaining a gradient map reflecting edge strength. Subsequently, an edge-weighted mask is generated based on this edge gradient map, with larger numerical weights at edge locations and smaller values ​​in flat areas.

[0056] Finally, the system spatially weights the cosine similarity calculated above based on the edge-weighted mask to construct the final orthogonal regularized loss function. The purpose of this step is to make the network focus its optimization on edge regions when calculating the total loss. Through this weighting, the orthogonality penalty weight of image edge regions is significantly higher than that of flat regions, forcing the network to increase the corresponding processing weights at the boundaries where errors are most likely to occur.

[0057] Example 5: In this embodiment, the step of performing channel splicing and fusion decoding on the first source private feature map, the second source private feature map, and the common feature map according to a preset feature recombination strategy includes: Global feature analysis is performed on the first source private feature map and the second source private feature map respectively to obtain the corresponding first source dynamic enhancement coefficient and second source dynamic enhancement coefficient; The first source private feature map is weighted according to the first source dynamic enhancement coefficient, and the second source private feature map is weighted according to the second source dynamic enhancement coefficient; The weighted first source private feature map, the weighted second source private feature map, and the shared feature map are concatenated along the channel dimension. The concatenated hybrid features are input into the decoder network, and multiple deconvolution operations are performed until the image size is restored to obtain the target fused image.

[0058] In this embodiment, before the shared feature map is included in the cascading, the following steps are also included: Obtain the pixel-wise maximum value images of the weighted first source private feature map and the weighted second source private feature map; The pixel-by-pixel maximum value map is superimposed on the common feature map to compensate for the intensity signal lost during the acquisition of redundant components.

[0059] After the gradient orthogonal projection and residual stripping operations described in the previous embodiment, the system has obtained three sets of feature components that are highly independent in mathematical statistics: the first source private feature map (e.g., infrared thermal radiation features), the second source private feature map (e.g., visible light texture features), and the common feature map (scene geometric structure features).

[0060] If these three sets of features are simply superimposed linearly, the differences in importance of different modes in the current specific scene may be ignored (for example, in a completely dark environment, the visible light private features may only contain noise), and the preceding subtraction stripping operation may cause the energy intensity of some bright targets to be attenuated. Therefore, this embodiment designs a reconstruction decoding scheme that includes dynamic weight allocation and signal compensation.

[0061] Specifically, before entering the final decoding stage, the system first performs global feature analysis on the first and second source private feature maps. This process enables it to automatically determine which modality's information is more critical based on the content of the input image. The system compresses the spatial dimension of each private feature map into a feature vector through a global average pooling operation. This vector highly summarizes the global information content and activation intensity of that modality. Subsequently, a multilayer perceptron (MLP) or fully connected layer is used to perform a nonlinear mapping on this feature vector to generate a scalar weight, namely a dynamic enhancement coefficient. This coefficient reflects the network's assessment of the current modality's confidence. For example, in a smoke-covered scene, the enhancement coefficient of infrared features will be automatically increased, while the coefficient of visible light features will be suppressed. Then, the system uses these two coefficients to perform weighted processing on the corresponding private feature maps.

[0062] Next, this embodiment introduces a compensation mechanism based on pixel-wise maximum values. In the preceding orthogonal projection process, to forcibly remove background interference, redundant components are eliminated through subtraction. While this mathematically guarantees orthogonality, physically it may weaken certain target signals that are inherently very strong and span two modes (such as a fiercely burning flame, which has both infrared heat and visible light brightness). To compensate for this potential energy loss, the system calculates the maximum value at each pixel position of the weighted first-source private feature map and the weighted second-source private feature map, generating a pixel-wise maximum value map. Then, this maximum value map is superimposed on the common feature map, thereby injecting the most significant peak signal in the private branch back into the common feature map, ensuring that the dynamic range and contrast of the fused image are not reduced due to the decoupling operation.

[0063] After weighting and compensation, the system concatenates the processed first-source private feature map, second-source private feature map, and compensated common feature map along the channel dimension, thereby stacking features of different attributes along the depth direction and preserving information from all dimensions. Finally, this concatenated hybrid high-dimensional tensor is input into the decoder network, which consists of a series of upsampling layers or deconvolutional layers. Its function is to reconstruct features, gradually restoring abstract semantic features to the resolution size of the original image, and finally outputting the target fused image.

[0064] Example 6: In this embodiment, the method further includes: Construct a deep neural network that includes a feature extractor, an orthogonal projection module, and a fusion decoder; Define a total loss function that includes an orthogonal regularization loss term and a reconstruction consistency loss term, wherein the reconstruction consistency loss term is used to constrain the first source private feature map and the shared feature map to restore the first source image, and is also used to constrain the second source private feature map and the shared feature map to restore the second source image; The parameters of the deep neural network are updated using the backpropagation algorithm based on the total loss function.

[0065] Understandably, deep neural networks logically consist of three stages: encoding, decoupling, and decoding. The feature extractor constitutes the encoding stage, composed of multiple cascaded convolutional neural network layers. It is responsible for mapping the input first source image (e.g., an infrared image) and second source image (e.g., a visible light image) from pixel space to a high-dimensional feature manifold space. The orthogonal projection module is a differentiable mathematical operation unit embedded in the middle layer of the network. It strictly follows the Schmitt orthogonalization formula in the aforementioned embodiment to perform vector projection and subtraction operations, used to clean and separate features in real time during forward propagation. The fusion decoder constitutes the decoding stage. Its structure is usually symmetrical to the feature extractor. It maps the decoupled and recombined feature tensors back to the image space through deconvolution or upsampling operations, generating the final fused image.

[0066] It is also understandable that, in order to train the above network, the core lies in defining a total loss function that includes an orthogonal regularization loss term and a reconstruction consistency loss term. The orthogonal regularization loss term is mainly responsible for constraining the independence of features, that is, ensuring that private features do not contain shared information. The reconstruction consistency loss term, which is the focus of this embodiment, is responsible for constraining the integrity and content fidelity of features. If there is only orthogonal constraint, the network may learn all features to zero in order to reduce the loss (the zero vector is orthogonal to any vector), resulting in feature collapse. The reconstruction consistency loss term forces the decoupled features to be able to restore the original image, thereby avoiding information loss.

[0067] More specifically, the loss term requires that the reconstructed image generated after the first source private feature map and the shared feature map are as close as possible to the original first source image after being processed by the decoder; similarly, the combination of the second source private feature map and the shared feature map should be able to restore the original second source image. This design is based on the concept of self-supervised learning, that is, using the input data itself as a label to supervise the training of the network.

[0068] Example 7: In this embodiment, the process further includes the following steps before acquiring the first source image and the second source image to be fused: Perform geometric registration on the acquired first and second source images; The registered image is normalized to a preset numerical range and adjusted to a uniform resolution size.

[0069] In practical multimodal imaging systems, the first source image (such as a long-wave infrared thermal image) and the second source image (such as a visible light image) are typically acquired by two physically independent optical sensors. Due to the physical distance (baseline) between the sensors and the difference in focal length of the optical lenses, parallax, rotational deviation, and scaling differences inevitably exist between the acquired raw images.

[0070] If an unregistered image is directly fed into the network, the feature locations in the first source image may be misaligned by tens of pixels with the feature locations in the second source image. In this case of misalignment, the spatial response intersection logic defined in subsequent embodiments will completely fail; the reason is that infrared features and visible light features cannot correspond spatially, and the product result will not correctly activate the common region, causing the network to be unable to extract the correct background contour.

[0071] Therefore, the system first performs geometric registration on the acquired first source image and second source image. This process usually selects one of the images with a wider field of view or higher resolution as the reference (usually a visible light image). It uses a feature point matching algorithm (such as SIFT or ORB) to extract the same feature point pairs in the two images and calculates the homography matrix or affine transformation matrix to map the image to be registered onto the coordinate system of the reference image.

[0072] Example 8: As attached Figure 2 As shown, this embodiment provides an image fusion system based on three-branch feature reconstruction and fusion, the system comprising: An image acquisition module is used to acquire a first source image and a second source image to be fused, and to map the first source image and the second source image to a high-dimensional feature space respectively to obtain an initial feature map, wherein the initial feature map includes a first source initial feature map and a second source initial feature map; A common feature extraction module is used to construct a common feature backbone branch based on the first source initial feature map and the second source initial feature map, and to obtain the intersection of the first source initial feature map and the second source initial feature map in spatial response, so as to extract a common feature map that represents the scene structure and background contour. A private feature extraction module is used to construct private branches, a first source residual private branch and a second source residual private branch according to the common feature map. In the private branches, the redundant components of the initial feature map in the direction of the common feature map are obtained, and the first source private feature map and the second source private feature map orthogonal to the common feature map are obtained according to the redundant components. The private branches include the first source private branch and the second source private branch. The fusion decoding module is used to perform channel splicing and fusion decoding on the first source private feature map, the second source private feature map and the common feature map according to a preset feature recombination strategy, so as to obtain a target fusion image containing the first source target information and the second source texture details.

[0073] In this embodiment, during actual operation, the image acquisition module first synchronously acquires the first source image (e.g., data output from a long-wave infrared thermal imager) and the second source image (e.g., data output from a visible light camera) to be fused via a hardware interface. Given the physical differences in field of view and resolution between different sensors, the image acquisition module integrates a geometric registration algorithm unit to perform homography transformation based on feature point matching, ensuring strict alignment of the two images in the pixel coordinate system. Subsequently, the module uses a pre-trained convolutional neural network as a feature extractor to map the first and second source images to a high-dimensional feature space. This high-dimensional feature space refers to the abstract data domain formed after multiple nonlinear transformations of the image data. In this domain, the original pixel grayscale values ​​are converted into feature response maps with hundreds of channels, each representing a specific semantic attribute such as edge, texture, or shape. Finally, the module outputs the first and second source initial feature maps, providing standardized data input for subsequent decoupling processing.

[0074] Secondly, the shared feature extraction module incorporates attention calculation logic based on spatial response intersection. Specifically, this module performs channel compression on the two initial input feature maps and executes element-wise multiplication. This operation logically simulates the "AND gate" mechanism in a circuit: the feature at a given location is activated and preserved only when both infrared and visible light features exhibit high response values ​​at the same spatial location (i.e., both detect the object's outline). Through this mechanism, the shared feature extraction module can accurately filter out noise or unique textures that only appear in a single modality, thereby outputting a clean shared feature map that represents the scene structure and background outline, ensuring that subsequent decoupling operations of the system have an accurate reference coordinate system.

[0075] Subsequently, the private feature extraction module is designed to completely eliminate the feature entanglement phenomenon commonly found in traditional fusion algorithms. The module internally constructs two parallel computational pipelines to process the first and second source residual private branches, respectively. During processing, the module treats the shared feature map output from the previous stage as a reference vector in a high-dimensional vector space and the initial feature map as the target vector. The module uses a built-in vector dot product operation unit to calculate the projection of the target vector onto the reference vector direction, i.e., the redundant component. This redundant component physically corresponds to the background contour information mixed in the initial features. Subsequently, the module performs a subtraction operation to physically remove this redundant component from the initial features. This mathematical operation forces the remaining feature components to be statistically perpendicular to the shared features, thus obtaining the first and second source private feature maps. These two private feature maps only contain information unique to each modality (such as high-heat targets in infrared or surface colors in visible light) and no longer contain any background contours, thereby eliminating the possibility of double contours or ghosting at the edges of the fused image at the source.

[0076] Finally, the fusion decoding module incorporates a preset feature recombination strategy, typically including dynamic weight allocation subunits and channel cascade subunits. The module first performs depth concatenation of the first-source private feature map, the second-source private feature map, and the shared feature map along the channel dimension, constructing a hybrid feature tensor containing all information. This tensor is then fed into the decoder network, which consists of a series of deconvolutional or upsampling layers, progressively restoring abstract high-dimensional semantic features into a high-resolution two-dimensional pixel matrix. During this process, the shared features provide a clear skeleton, the first-source private features fill in the thermal radiation target, and the second-source private features fill in surface texture details, ultimately generating a fused target image with both high target contrast and rich background detail.

[0077] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0078] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0079] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a multimedia terminal device (which may be a mobile phone, computer, television receiver, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0080] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. An image fusion method based on three-branch feature reconstruction and fusion, characterized in that, The method includes the following steps: A first source image and a second source image to be fused are obtained, and the first source image and the second source image are respectively mapped to a high-dimensional feature space to obtain an initial feature map, wherein the initial feature map includes a first source initial feature map and a second source initial feature map; Based on the first source initial feature map and the second source initial feature map, a common feature backbone branch is constructed, and the intersection of the first source initial feature map and the second source initial feature map in spatial response is obtained to extract the common feature map representing the scene structure and background contour. Private branches are constructed based on the common feature map. In the private branches, redundant components of the initial feature map in the direction of the common feature map are obtained. Based on the redundant components, a first source private feature map and a second source private feature map orthogonal to the common feature map are obtained. The private branches include a first source private branch and a second source private branch. According to a preset feature recombination strategy, the first source private feature map, the second source private feature map, and the common feature map are channel-stitched and fused to obtain a target fused image containing the first source target information and the second source texture details.

2. The image fusion method based on three-branch feature reconstruction and fusion as described in claim 1, characterized in that, The step of obtaining the intersection of the first source initial feature map and the second source initial feature map in spatial response to extract a common feature map representing the scene structure and background contour includes: The first source initial feature map and the second source initial feature map are compressed in the channel dimension to obtain the first source space descriptor and the second source space descriptor. Element-wise multiplication is performed on the first source spatial descriptor and the second source spatial descriptor to enhance the common response region and suppress the unilateral response region, and to obtain the spatial attention mask. The linear combination of the first source initial feature map and the second source initial feature map is weighted and filtered according to the spatial attention mask to obtain the common feature map.

3. The image fusion method based on three-branch feature reconstruction and fusion as described in claim 1, characterized in that, The step of obtaining the redundant components of the initial feature map in the direction of the common feature map in the private branch, and obtaining the first source private feature map and the second source private feature map orthogonal to the common feature map based on the redundant components, includes: The first source initial feature map is regarded as the target vector in the vector space, and the common feature map is regarded as the reference vector; Obtain the inner product between the target vector and the reference vector, and normalize the inner product according to the magnitude of the reference vector to obtain the first projection coefficient; Multiply the first projection coefficient by the reference vector to obtain the redundant component of the first source initial feature map in the direction of the common feature map; The redundant component is subtracted from the first source initial feature map to obtain the first source private feature map, such that the first source private feature map is mathematically perpendicular to the common feature map.

4. The image fusion method based on three-branch feature reconstruction and fusion as described in claim 3, characterized in that, The steps for obtaining the second source private feature map are the same as those for obtaining the first source private feature map.

5. The image fusion method based on three-branch feature reconstruction and fusion as described in claim 1, characterized in that, The process of constructing private branches based on the shared feature map also includes: Obtain the cosine similarity between the first source private feature map and the shared feature map, and the cosine similarity between the second source private feature map and the shared feature map; Extract edge gradient maps from the first source image or the second source image, and obtain an edge weighted mask based on the edge gradient maps; The cosine similarity is spatially weighted according to the edge-weighted mask so that the orthogonality penalty weight of the image edge region is higher than that of the flat region.

6. The image fusion method based on three-branch feature reconstruction and fusion as described in claim 1, characterized in that, The step of performing channel splicing and fusion decoding on the first source private feature map, the second source private feature map, and the common feature map according to a preset feature recombination strategy includes: Global feature analysis is performed on the first source private feature map and the second source private feature map respectively to obtain the corresponding first source dynamic enhancement coefficient and second source dynamic enhancement coefficient. The first source private feature map is weighted according to the first source dynamic enhancement coefficient, and the second source private feature map is weighted according to the second source dynamic enhancement coefficient; The weighted first source private feature map, the weighted second source private feature map, and the shared feature map are concatenated along the channel dimension. The concatenated hybrid features are input into the decoder network, and multiple deconvolution operations are performed until the image size is restored to obtain the target fused image.

7. The image fusion method based on three-branch feature reconstruction and fusion as described in claim 6, characterized in that, Before incorporating the shared feature map into the cascade, the following is also included: Obtain the pixel-wise maximum value images of the weighted first source private feature map and the weighted second source private feature map; The pixel-by-pixel maximum value map is superimposed on the common feature map to compensate for the intensity signal lost during the acquisition of redundant components.

8. The image fusion method based on three-branch feature reconstruction and fusion as described in claim 1, characterized in that, The method further includes: Construct a deep neural network that includes a feature extractor, an orthogonal projection module, and a fusion decoder; Define a total loss function that includes an orthogonal regularization loss term and a reconstruction consistency loss term, wherein the reconstruction consistency loss term is used to constrain the first source private feature map and the shared feature map to restore the first source image, and is also used to constrain the second source private feature map and the shared feature map to restore the second source image; The parameters of the deep neural network are updated using the backpropagation algorithm based on the total loss function.

9. The image fusion method based on three-branch feature reconstruction and fusion as described in claim 1, characterized in that, Before acquiring the first and second source images to be fused, the process also includes: Perform geometric registration on the acquired first and second source images; The registered image is normalized to a preset numerical range and adjusted to a uniform resolution size.

10. An image fusion system based on three-branch feature reconstruction and fusion, characterized in that, The system includes: An image acquisition module is used to acquire a first source image and a second source image to be fused, and to map the first source image and the second source image to a high-dimensional feature space respectively to obtain an initial feature map, wherein the initial feature map includes a first source initial feature map and a second source initial feature map; A common feature extraction module is used to construct a common feature backbone branch based on the first source initial feature map and the second source initial feature map, and to obtain the intersection of the first source initial feature map and the second source initial feature map in spatial response, so as to extract a common feature map that represents the scene structure and background contour. A private feature extraction module is used to construct private branches, a first source residual private branch and a second source residual private branch according to the common feature map. In the private branches, the redundant components of the initial feature map in the direction of the common feature map are obtained, and the first source private feature map and the second source private feature map orthogonal to the common feature map are obtained according to the redundant components. The private branches include the first source private branch and the second source private branch. The fusion decoding module is used to perform channel splicing and fusion decoding on the first source private feature map, the second source private feature map and the common feature map according to a preset feature recombination strategy, so as to obtain a target fusion image containing the first source target information and the second source texture details.

Images