A multi-modal feature fusion based harsh environment image enhancement and restoration method

Abstract

The application provides a kind of multi-modal feature fusion-based harsh environment image enhancement and restoration method, belongs to image enhancement processing field;The method first obtains multi-modal image and calculates each modal degradation degree graph;Based on the degradation degree and gradient amplitude, a fusion weight is constructed, and the local dominant direction and coherence are obtained by structure tensor analysis;According to the coherence, the pixels are partitioned, and the adaptive processing of each modal gradient is carried out by using hard projection, isotropic fusion or soft projection strategy respectively;Based on the fusion weight, the processed gradient is weighted and fused, and the fusion gradient field is obtained combined with the enhancement factor g (x) ;Finally, an energy function containing pixel fidelity term and gradient fidelity term is constructed, and the enhanced fusion image is reconstructed by weighted least squares optimization;The application suppresses the contribution of blurred modal through degradation perception mechanism, and solves the gradient direction conflict through structure adaptive projection, which significantly improves the clarity and structure fidelity of multi-modal image fusion in harsh environment.

Description

Technical Field

[0001] This invention relates to the field of image enhancement processing, and in particular to a method for enhancing and restoring images in harsh environments based on multimodal feature fusion. Background Technology

[0002] Multimodal image fusion technology aims to effectively integrate image information acquired by different sensors or imaging devices in the same scene to generate a more information-rich and comprehensive fused image, which has broad application prospects in military reconnaissance, security monitoring, remote sensing, medical imaging, and autonomous driving. Especially under harsh environmental conditions, such as fog, low light, rain, snow, and underwater scenarios, single-modal images often suffer severe information loss due to environmental degradation. For example, visible light images experience decreased contrast and blurred details; infrared images, while unaffected by lighting, lack texture information and have high noise levels; and polarized images can provide material information but have limited signal-to-noise ratios. Therefore, fusing complementary information from multimodal images to achieve image enhancement and restoration in harsh environments has become a research hotspot in this field.

[0003] Currently, mainstream image fusion methods can be mainly divided into the following categories: One category is pixel-level fusion methods based on the spatial domain or transform domain, such as weighted average, principal component analysis (PCA), and fusion methods based on multi-scale decomposition (such as Laplacian pyramid, wavelet transform, and contourlet transform). These methods typically decompose the source image and then select or fuse coefficients at each scale according to specific rules (such as taking the largest absolute value or the largest energy), before reconstructing the image through inverse transform. However, when processing images from harsh environments, these methods often ignore the differences in the degree of degradation in local regions of different modalities, simply treating the coefficients of clear regions and blurry regions equally. This results in residual blurry components in the fusion result, weakened clear structures, and problems such as blurred edges and loss of detail.

[0004] Another type of fusion method is based on sparse representation or low-rank decomposition. This type of method assumes that the image can be sparsely represented by an overcomplete dictionary, and then extracts common and unique components through joint sparse coding or low-rank matrix factorization for fusion. Although it can separate noise and structure to some extent, the dictionary construction and sparse coding process are computationally complex, and its adaptability to complex and varied degradation patterns in harsh environments is limited, making it difficult to accurately model the degradation process of each modality.

[0005] In recent years, deep learning-based fusion methods have developed rapidly, utilizing convolutional neural networks (CNNs) or generative adversarial networks (GANs) to automatically learn fusion rules and feature representations, achieving good subjective visual effects. However, these methods typically rely on large-scale paired training data, making it extremely difficult to obtain ideal full-reference images in real-world harsh environments. Furthermore, the black-box nature of the models leads to poor interpretability, and the fusion process lacks physical prior constraints. When encountering degradation types not covered in the training set, the fusion results may exhibit artifacts or structural distortions.

[0006] Furthermore, optimization-based fusion methods model the fusion problem as an energy minimization problem, solving for the target image by designing data terms and regularization constraints. However, existing optimization methods typically employ simple maximization or weighted averaging strategies when constructing the fusion gradient field, failing to adequately consider the consistency of gradient directions across different modes. When different modes have opposite gradient directions at the same location or when there is a large angle between them (e.g., visible light and infrared images produce opposite gradients at object edges due to different imaging mechanisms), simple averaging can lead to gradient cancellation, resulting in weakened edge intensity or even structural ghosting.

[0007] In summary, existing technologies generally suffer from the following technical bottlenecks when dealing with multimodal image fusion in harsh environments: lack of quantitative perception and utilization of the degree of modal differentiation degradation; gradient domain fusion strategies fail to effectively solve the problem of directional conflict; and it is difficult to balance the structural fidelity and detail enhancement capabilities of the fusion results.

[0008] Therefore, how to realize a multimodal image enhancement and restoration method that can adaptively perceive the degree of degradation, maintain the consistency of gradient direction, and have global structural coherence is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method for image enhancement and restoration in harsh environments based on multimodal feature fusion. It aims to solve the technical problems of structural distortion, loss of detail, and insufficient robustness in multimodal image fusion under harsh environments by combining degradation perception and structural adaptive gradient fusion with weighted least squares optimization and organically combining degradation perception, structural adaptive gradient fusion and global optimization. Compared with existing fusion methods, it has achieved significant progress in image clarity, structural fidelity, environmental adaptability and computational stability.

[0010] To achieve the above objectives, this application proposes a method for image enhancement and restoration in harsh environments based on multimodal feature fusion, comprising the following steps: Step A: Acquire N modal images of the same scene, as the multimodal image I to be processed. iWhere i represents the corresponding modal image change, i = 1, 2, ..., N, N ≥ 2; Step B: Based on the pre-trained first neural network model DegNet, obtain and calculate the I of each modality image. i Degradation level diagram D i (x); where the first neural network model DegNet takes a multimodal image patch as input and outputs a corresponding degradation level map D. i (x); the degradation level diagram D i (x) is used to characterize the degree of degradation at each pixel position x in the image; Step C: Based on the degradation level map D i (x) and the gradient field of each modality image I i (x), calculate the fusion gradient field G(x); Step D: Construct an energy function E(F) for the enhanced fused image F to be solved, the energy function including a pixel fidelity term and a gradient fidelity term; wherein, the pixel fidelity term constrains the enhanced fused image F and the reference image I. ref The gradient fidelity term constrains the difference between the gradient field of the enhanced fused image F and the fused gradient field G(x); Step E: The enhanced fused image F is obtained by minimizing the energy function E(F).

[0011] As a further solution, the first neural network model DegNet employs different prior methods to obtain degradation maps for training; wherein, the prior methods include: Degradation of visible light images in foggy weather is estimated based on transmittance estimation using dark channel priors. The degree of degradation of infrared images is estimated based on the noise level estimation using local standard deviation. The degradation level of low-light images is estimated based on the mean available brightness.

[0012] As a further solution, the calculation of the fused gradient field G(x) in step C specifically includes: Step C1: According to the degradation degree diagram D i (x) and gradient magnitude | I i (x)∣, calculate the fusion weights for each modality. Among them, the higher the degree of degradation, the smaller the weight; Step C2: Gradient field for each mode I i (x) is weighted and averaged to obtain the weighted average gradient. ; Step C3: Based on the degradation degree of the local clearest mode min i D i (x), calculate the enhancement factor g(x), which is negatively correlated with the degree of degradation of the clearest mode; Step C4: Apply the enhancement factor g(x) to the weighted average gradient. I avg (x), thus obtaining the fused gradient field G(x) = g(x). I avg (x).

[0013] As a further solution, the fusion weights described in step C1 The calculation formula is: Where α is a weight control parameter greater than 0, ε and δ are constants to prevent division by zero, and j is the summation iteration variable.

[0014] As a further solution, the formula for calculating the enhancement factor g(x) in step C3 is as follows: Where β is the enhancement amplitude control parameter greater than 0, and γ is the decay rate control parameter greater than 0.

[0015] As a further solution, the specific form of the energy function E(F) in step D is: Where p is the pixel index, N(p) represents the neighborhood of pixel p, and G pq To fuse the projection of the gradient field G(x) onto the direction from pixel p to its neighboring pixel q, λ is a parameter balancing the pixel fidelity term and the gradient fidelity term; this is achieved by solving the corresponding linear equation system (I+λD). D)F=I ref +λD G yields the enhanced fused image F, where I is the identity matrix and D is the discrete difference matrix.

[0016] As a further solution, the calculation of the fused gradient field G(x) in step C also includes the step of constraining the orientation consistency: Step F1: Based on the gradient field of each modality image I i (x), calculate the local structure tensor and obtain the global dominant direction e. max (x); Step F2: Convert the gradients of each mode I i(x) Projected onto the global dominant direction e max On (x), the projected gradient is obtained; Step F3: Based on the degradation level map D i (x) and gradient magnitude | I i (x)∣, weighted fusion of the projected gradients yields a fused gradient field with consistent direction.

[0017] As a further solution, the global dominant direction e described in step F1 max (x) is obtained by the following method: for the structure tensor S of each mode. i (x) is weighted and averaged to obtain the weighted average structure tensor. The weights are the fusion weights. ; for the weighted average structure tensor Perform eigenvalue decomposition and take the eigenvector corresponding to the largest eigenvalue as the global dominant direction e. max (x).

[0018] As a further solution, the calculation of the fused gradient field G(x) in step C also includes a step of adaptive structural projection, replacing step F2: Step G1: For the weighted average structure tensor Perform eigenvalue decomposition and calculate the local coherence Coh(x) based on the eigenvalues ​​obtained from the decomposition. Step G2: Divide the pixels into different types of regions according to the coherence Coh(x), and apply a strategy that matches the different types of regions to the modal gradient. I i (x) is processed to obtain the processed gradient. I i type (x); Step G3: Based on the degradation level map D i (x) and gradient magnitude | I i (x)∣, for the processed gradient I i type (x) is weighted and fused.

[0019] As a further solution, the region types mentioned in step G2 include high coherence regions, low coherence regions, and medium coherence regions; For regions of high coherence, the processing strategy is to project the gradient onto the globally dominant direction e. max On (x), the projected gradient is obtained; For regions with low coherence, the processing strategy is to preserve the original gradient. I i (x); For regions of moderate coherence, the processing strategy is soft projection, which simultaneously preserves the dominant directional component and a portion of the vertical directional component to obtain the soft-projected gradient image. I i soft (x).

[0020] Compared with related technologies, the image enhancement and restoration method for harsh environments based on multimodal feature fusion provided by this invention has the following advantages: 1. This invention effectively suppresses the negative impact of severely degraded modes on the fusion result by constructing degradation-aware fusion weights, thus avoiding the transmission of blurry components. Simultaneously, by introducing directional consistency constraints and a structural adaptive projection strategy in the gradient domain, it solves the gradient direction conflict problem caused by differences in imaging mechanisms in multimodal images, eliminating edge ghosting and structural blurring phenomena easily generated by traditional weighted averaging methods. Furthermore, by combining a weighted least squares global optimization framework, the final enhanced fused image accurately approximates the fusion gradient field with enhanced structural information while maintaining pixel value fidelity, significantly improving the edge sharpness, texture detail, and overall visual clarity of the image.

[0021] 2. This invention achieves fine quantification of the degradation degree of different modalities and regions by calculating pixel-level degradation degree maps. Based on this, the enhancement factor g(x) is dynamically calculated according to the degradation degree of the locally clearest modality, realizing adaptive detail compensation for blurred regions, while avoiding over-enhancement and noise introduction in clear regions. Furthermore, by introducing a structure-adaptive projection mechanism, it can distinguish edge regions, flat regions, and texture regions according to local coherence, and adopt hard projection, isotropic fusion, or soft projection strategies respectively, so that the fusion process can adapt to the local structural features of the image, enhancing edges while avoiding noise amplification in flat regions and information loss in texture regions.

[0022] 3. The core framework of this invention is based on explicit mathematical models and numerical optimization methods, without relying on large-scale training data and black-box models, and has a good theoretical foundation and interpretability. On this basis, by introducing a lightweight neural network module to assist in degradation degree estimation and parameter prediction, the stability of the mathematical optimization framework is preserved, and the algorithm is endowed with the ability to learn complex degradation patterns from data. The DegNet module can overcome the failure problem of traditional prior methods (such as dark channel priors) in specific scenarios (such as white objects in foggy weather), and provide more accurate degradation degree estimation.

[0023] 4. This invention unifies pixel fidelity and gradient fidelity terms within a weighted least squares optimization framework, obtaining the global optimal solution by solving a large system of sparse linear equations. This global optimization strategy avoids block artifacts, discontinuities, or other problems that may be introduced by step-by-step processing or local operations, ensuring the overall structural coherence and visual naturalness of the enhanced fused image. Compared with fusion methods based on local rules, the results of this invention are smoother and more natural in terms of image brightness, contrast, and color (if applicable), conforming to the characteristics of human visual perception. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0026] Figure 1 A schematic diagram of a process for enhancing and restoring harsh environment images based on multimodal feature fusion provided by the present invention; The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0028] Example 1 Please see Figure 1 This embodiment provides a method for image enhancement and restoration in harsh environments based on multimodal feature fusion, including the following steps: Step A: Acquire N modal images of the same scene, as the multimodal image I to be processed. i Where i represents the corresponding modal image change, i = 1, 2, ..., N, N ≥ 2; Step B: Based on the pre-trained first neural network model DegNet, obtain and calculate the I of each modality image. i Degradation level diagram D i(x); where the first neural network model DegNet takes a multimodal image patch as input and outputs a corresponding degradation level map D. i (x); the degradation level diagram D i (x) is used to characterize the degree of degradation at each pixel position x in the image; Step C: Based on the degradation level map D i (x) and the gradient field of each modality image I i (x), calculate the fusion gradient field G(x); Step D: Construct an energy function E(F) for the enhanced fused image F to be solved, the energy function including a pixel fidelity term and a gradient fidelity term; wherein, the pixel fidelity term constrains the enhanced fused image F and the reference image I. ref The gradient fidelity term constrains the difference between the gradient field of the enhanced fused image F and the fused gradient field G(x); Step E: The enhanced fused image F is obtained by minimizing the energy function E(F).

[0029] It should be noted that in harsh environments (such as smog, low light, underwater, etc.), multimodal images (visible light, infrared, polarization, etc.) are often degraded to varying degrees, resulting in low image contrast, blurred details, and amplified noise. Existing fusion methods usually directly perform weighted averaging of pixels or features, or use simple multi-scale decomposition followed by fusion, but they generally ignore the differences in the degree of degradation of each modality and the impact of degradation on structural information.

[0030] For example, in foggy weather, visible light images have blurred edges, while infrared images have clear edges but little texture. Simple weighted averaging can cause severely degraded modes to contribute too much blurring to the fusion result, while the structural information of clear modes is weakened, resulting in edge ghosting, texture loss, or noise amplification. This problem stems from a lack of quantitative perception of the degree of degradation and constraints on structural consistency, preventing the fusion result from effectively utilizing the complementary advantages of multiple modes.

[0031] To address this, this embodiment proposes a multimodal image enhancement and restoration method that combines degradation estimation, adaptive gradient fusion, and global optimization. The core idea is to estimate the degradation degree of each modality through prior knowledge and construct a fusion gradient field accordingly. At the same time, weighted least squares optimization is introduced so that the gradient of the final image approximates the fusion gradient field while maintaining pixel consistency, thereby achieving structure preservation and detail enhancement.

[0032] Furthermore, in the degradation-aware gradient fusion and weighted least squares optimization, the fused gradient field G(x) is usually obtained by weighted averaging of the gradients of each modality, with the weights based on the degree of degradation and the magnitude of the local gradient. While this weighted averaging strategy can suppress the contribution of degraded modes based on sharpness, it ignores the potential conflicts between gradient directions of different modes.

[0033] When two or more modalities have gradients with opposite directions or significant angles at the same pixel, simple weighted averaging can lead to a reduction in the magnitude of the synthesized gradient or even their cancellation, thereby weakening structural information such as edges and textures. Especially in harsh environments, different modalities may produce gradient responses with opposite directions at the edges of the same object due to their different imaging mechanisms (such as visible light and infrared). Direct averaging can cause structural blurring or ghosting in the fused image, which defeats the purpose of enhancement and restoration.

[0034] To address the aforementioned issues, this embodiment further introduces gradient direction consistency constraints while maintaining the degradation perception weights. This allows the fused gradient field to enhance the sharp regions while preserving the geometric consistency of the structural information of each modality, avoiding reverse cancellation, and thus obtaining sharper, more natural edges and details.

[0035] Specifically, this embodiment extends the gradient fusion process to two steps: first, the structure tensor of each modality is calculated to estimate the local dominant gradient direction; then, the gradients of each modality are subjected to adaptive projection and weighted fusion to ensure that the synthesized gradient is consistent with the dominant direction and retains its magnitude. The improved fused gradient field is used as the optimization objective, and the final image is still solved by weighted least squares to maintain global structural coherence.

[0036] Furthermore, for the input multimodal image I i (i=1,2,…,N), the degradation degree map D is calculated using a priori method suitable for its degradation type. i (x). For example: For visible light images in foggy conditions, the transmittance t(x) is estimated using the dark channel prior, then the fog concentration D... vis (x)=1 t(x), normalized to [0,1].

[0037] For infrared images, the noise level can be estimated using the local standard deviation, and then normalized.

[0038] For low-light images, the degree of darkness can be estimated using the mean brightness.

[0039] Degradation level diagram D i The larger the value of (x), the more severe the degradation of the region and the lower the credibility of the information.

[0040] Degradation level diagram D in the original scheme i(x) Relies on prior methods (such as dark channel, local standard deviation). These priors may fail in some complex scenarios (such as white objects in foggy weather, non-Gaussian infrared noise). The degradation level of each pixel can be directly estimated using AI models (such as deep regression networks), with a multimodal image as input and a degradation map as output. Synthetic degradation data or real paired data can be used during training. The estimated D... i (x) is still used for subsequent weight calculations and enhancement factor g(x), keeping the mathematical framework unchanged.

[0041] In one specific embodiment, a simulation dataset (containing multimodal images and corresponding real degradation maps) is constructed to train DegNet; ParamNet is trained through reinforcement learning using a no-reference quality evaluation metric.

[0042] AI Module Design: Degradation Estimation Network (DegNet) Input: Multimodal image patch (e.g., 64×64 pixels, containing visible light, infrared, polarization, etc., with N channels).

[0043] Output: Degradation level D for each pixel i (x)∈[0,1], with the same resolution as the input image.

[0044] Network structure: A lightweight fully convolutional network (such as a simplified version of U-Net). The encoder uses 3-4 convolutional layers (stride 2) to extract features, and the decoder uses deconvolution or upsampling + convolution to restore resolution. Skip connections preserve details. The output layer uses Sigmoid activation.

[0045] Training data generation: Samples are selected from a clear multimodal image dataset (such as RGB+infrared pairing).

[0046] Individual simulation degradation is applied to each mode: for example, adding fog to visible light (using depth maps to generate transmittance), adding noise (Gaussian / Poisson), and adding low light (reducing brightness and increasing noise); adding non-uniform noise and blurring to infrared images.

[0047] Record the degree of degradation applied as the true value: for example, fog concentration can be represented by the reciprocal of transmittance, and noise level can be represented by the local standard deviation, normalized to [0,1].

[0048] Generate a large number of image patch pairs (I deg D true ).

[0049] Loss function: Mean squared error (MSE) or mean absolute error (MAE) measures the difference between the predicted degraded map and the ground truth map. Edge preservation terms (such as gradient loss) can be added to improve the accuracy of the degraded map at edges.

[0050] Inference stage: The entire image is divided into blocks and input into DegNet, which are then stitched together to obtain a full-resolution degradation map D. i (x).

[0051] Furthermore, the calculation of the fused gradient field G(x) in step C specifically includes: Step C1: According to the degradation degree diagram D i (x) and gradient magnitude | I i (x)∣, calculate the fusion weights for each modality. Among them, the higher the degree of degradation, the smaller the weight; Step C2: Gradient field for each mode I i (x) is weighted and averaged to obtain the weighted average gradient. ; Step C3: Based on the degradation degree of the local clearest mode min i D i (x), calculate the enhancement factor g(x), which is negatively correlated with the degree of degradation of the clearest mode; Step C4: Apply the enhancement factor g(x) to the weighted average gradient. I avg (x), thus obtaining the fused gradient field G(x) = g(x). I avg (x).

[0052] It's important to note that in multimodal images, the quality of different modalities at the same pixel location varies. For example, in foggy weather, visible light images have blurred edges, while infrared images have sharp edges. If both are treated equally, the blurred edges will contaminate the fusion result.

[0053] Therefore, this embodiment proposes a weight calculation formula that combines the degree of degradation and the gradient magnitude, wherein the fusion weight described in step C1... The calculation formula is: Where α is a weight control parameter greater than 0, ε and δ are constants to prevent division by zero, and j is the summation iteration variable.

[0054] The core degradation sensing mechanism of this formula originates from the degradation inhibition term exp( αD i (x)); where D i The larger the (x) value (the more severe the degradation), the smaller the value of the degradation suppression term, thus reducing the weight of that modality in the current pixel. The parameter α controls the strength of the suppression; the larger α is, the more significant the impact of degradation on the weights.

[0055] The structural retention term of this formula (∣) I i (x)∣+ε): Gradient magnitude∣ I i (x)∣ reflects the local structural strength of the image (such as edges and textures). The larger the magnitude, the richer the structural information at that point, and the higher the weight should be assigned. ε is a small positive constant used to prevent regions with zero gradients from also having zero weights, ensuring numerical stability.

[0056] The normalization denominator of this formula sums the weighted values ​​of all modes and then normalizes them to ensure that the sum of the weights of all modes is 1 (with a very small amount of δ added to prevent division by zero).

[0057] The weight design in this embodiment implements a "survival of the fittest" principle: modes with less degradation and stronger structure contribute more, while modes with more degradation and weaker structure contribute less. This forms the basis for all subsequent gradient fusion operations, ensuring fusion quality from the outset.

[0058] Step C2 involves initially fusing the gradient information of each mode based on weights, and obtaining the weighted average gradient by weighting the gradient fields of each mode. This initially solved the problem of fuzzy propagation.

[0059] However, the initial fusion gradient I avg (x) While it suppresses blur, it does not actively enhance image sharpness. In harsh environments, we hope to further highlight blurred details without amplifying noise.

[0060] Therefore, step C3 calculates an enhancement factor g(x) based on the degree of degradation of the locally clearest mode. The formula for calculating the enhancement factor g(x) in step C3 is as follows: Where β is the enhancement amplitude control parameter greater than 0, and γ is the decay rate control parameter greater than 0.

[0061] The principle can be explained as follows: among all modes, take the value with the smallest degree of degradation (miniD). i (x)). This value represents the best quality of information available at that pixel location. This value will be large if all modalities are severely degraded (such as in deep fog); this value will be small if at least one modality is sharp.

[0062] g(x) is a value greater than or equal to 1. When miniD i When (x) is small (i.e., a clear mode exists), exp( γminiD iWhen (x) approaches 1, g(x) approaches 1+β, achieving a strong enhancement; when miniD i When (x) is large (all modes are blurred), the exponential term is close to 0 and g(x) is close to 1, resulting in a small enhancement effect, thus avoiding blindly enhancing noise.

[0063] Therefore, step C3 achieves adaptive enhancement: enhancement is performed only in areas where details can be recovered (at least one modality is sharp), while areas where all modalities are degraded are handled with care, thereby improving overall sharpness while effectively controlling the amplification of noise.

[0064] Step C4 ultimately yields an ideal gradient field that integrates multimodal information, suppresses degenerate modes, and enhances details in sharp regions. This gradient field will serve as the "target" for subsequent global optimization, guiding the generation of the final image.

[0065] In a more specific embodiment, suppose we are processing a pair of foggy day images: Visible Light Image I vis and infrared image I ir .

[0066] At the edge of a building: D vis High (with foggy blur), | I vis | Medium; D ir Lower (infrared radiation is not affected by fog), | I ir | Very high (sharp edges).

[0067] Result of step C1: Much larger .

[0068] Result of step C2: I avg (x) is mainly contributed by the infrared gradient, and the edge direction is correct.

[0069] Step C3 result: miniD i (x)=D ir It is very small, g is close to 1+β, and the enhancement factor g(x) is large.

[0070] Step C4 result: The final G is a strong edge with accurate orientation and significantly enhanced amplitude. Subsequent reconstructed images will have clear and sharp building edges.

[0071] And in a uniform sky covered by thick fog: D vis Very high, | I vis | Approaching 0, D ir It could also be higher (no texture, but with noise), | Iir | Smaller.

[0072] Step C1 result: All modal weights are small and similar.

[0073] Result of step C2: I avg (x) is a weak gradient close to 0.

[0074] Step C3 result: miniD i When x is large, g is close to 1, and the enhancement factor g(x) is small.

[0075] Step C4 result: The final G is a weak gradient field. Subsequent reconstructed images remain smooth in this region, avoiding the enhancement of infrared noise as texture, thus suppressing noise.

[0076] It's important to note that the steps described above construct an ideal fused gradient field G(x), which contains the gradient information we expect the final image to possess (such as sharp edges and enhanced details). However, the gradient field itself cannot be directly reconstructed into an image because reconstructing an image from the gradient field is an integral process with arbitrary constant offsets, and simple integration may lead to image brightness distortion or structural inconsistencies. We need a method to generate an image that approximates both the original multimodal information in pixel values ​​and our carefully designed target gradient field in gradients.

[0077] Furthermore, step D is applied to unify the two objectives of "pixel value consistency" and "gradient consistency" into a single mathematical optimization framework. An energy function E(F) with two constraints is proposed, and minimizing this function yields the globally optimal enhanced fused image. The specific form of the energy function E(F) is as follows: Where p is the pixel index, N(p) represents the neighborhood of pixel p, and G pq To fuse the projection of the gradient field G(x) onto the direction from pixel p to its neighboring pixel q, λ is a parameter balancing the pixel fidelity and gradient fidelity terms. The energy function consists of two parts: the sum of the pixel fidelity term (left-hand side) and the gradient fidelity term (right-hand side), connected by the balancing parameter λ. This is a typical regularized optimization framework aimed at finding a "compromise" solution that is as good as possible under both constraints.

[0078] In this embodiment, by minimizing this energy function, we "guide" the local difference relationship of the final image to approximate the pre-designed ideal gradient field. To this end, the energy function is written in matrix-vector form, transforming it into a linear least squares problem, and its corresponding normal equation is derived: This is a convex quadratic function with respect to F. Its minimum point can be obtained by setting the derivative (gradient) to zero. By differentiating E(F) with respect to F and setting it to zero, we can derive a system of linear equations: (I+λD D)F=I ref +λD G The enhanced fused image F is obtained, where I is the identity matrix, D is the discrete difference matrix, and Dij is the discrete difference matrix. D is a sparse matrix that corresponds to the Laplacian operator of the image in numerical computation, (I+λD) D) is a large sparse symmetric positive definite matrix, which guarantees that the system of equations has a unique solution.

[0079] Specifically, the matrix F obtained by solving this system of equations, although enormous (the same as the number of pixels in the image), is due to (I+λD) D) is sparse and can be solved quickly using efficient iterative algorithms (such as the conjugate gradient method), without the need to store or directly invert a large matrix.

[0080] Example 2 This embodiment, based on Embodiment 1, further defines and improves step C (calculating the fused gradient field G(x)). Addressing the gradient direction conflict problem that may arise from the simple weighted averaging in Embodiment 1, it introduces a direction consistency constraint. By considering the directional relationships of the gradients of each mode during the gradient fusion process, it ensures that the fused gradient field maintains geometric consistency, avoiding edge cancellation and structural ambiguity.

[0081] Specifically, step C, which involves calculating the fused gradient field G(x), also includes the step of constraining directional consistency. Step F1: Based on the gradient field of each modality image I i (x), calculate the local structure tensor and obtain the global dominant direction e. max (x); Step F2: Convert the gradients of each mode I i (x) Projected onto the global dominant direction e max On (x), the projected gradient is obtained; Step F3: Based on the degradation level map D i (x) and gradient magnitude | I i (x)∣, weighted fusion of the projected gradients yields a fused gradient field with consistent direction.

[0082] In the scheme of Example 1, the gradient field is fused. I avg(x) is obtained by analyzing the gradients of each mode. I i (x) is obtained by weighted averaging. This approach implicitly assumes that the gradients of each modality are in the same direction. However, in real multimodal imaging, this assumption often does not hold: Differences in imaging mechanisms: Visible light images reflect reflectivity, while infrared images reflect thermal radiation. The two may produce gradient responses in opposite directions at the edge of the same object.

[0083] Noise interference: Severely degraded modes may generate random gradient directions, interfering with the fusion results.

[0084] Registration error: Even after registration, a small offset can cause inconsistent gradient directions.

[0085] When the gradient directions of multiple modalities at the same pixel are opposite or have a large angle, simple weighted averaging will lead to a decrease in the magnitude of the synthesized gradient or even mutual cancellation, resulting in weakened edge intensity, blurred structure or ghosting.

[0086] To address this, this embodiment introduces the mathematical tool of structure tensor to statistically aggregate gradient information within a local window.

[0087] For each pixel x, the structure tensor S of each modal gradient is computed within a window W (e.g., 3×3 or 5×5) centered on x. i (x): Then, the structure tensors of each mode are weighted and averaged to obtain the weighted average structure tensor. :in, The fusion weights are defined in Example 1 and are based on the degree of degradation and the magnitude of the gradient.

[0088] For weighted average structure tensor Perform eigenvalue decomposition to obtain eigenvalues ​​λ1≥λ2≥0 and corresponding eigenvectors e1, e2. Take the eigenvector corresponding to the largest eigenvalue as the global dominant direction e. max (x)=e1.

[0089] The structure tensor is essentially a covariance matrix of a local gradient distribution. Its two eigenvectors represent the principal direction of the local gradient and the direction perpendicular to the principal direction, respectively, and the corresponding eigenvalues ​​reflect the gradient strength along these two directions.

[0090] The same fusion weights as in Example 1 were used when calculating the weighted average structure tensor. This means that modes with less degradation and stronger structure contribute more to the estimation of the dominant orientation, while the contribution of heavily degraded modes is suppressed, thereby improving the robustness of orientation estimation.

[0091] e max (x) represents the direction in which the gradient energy is most concentrated within the neighborhood of this pixel after considering all modes. This direction is most likely to correspond to the actual physical edge direction.

[0092] Therefore, this embodiment aggregates local gradient information through structural tensors and introduces degradation-aware weights. Step F1 can robustly estimate a dominant direction that can represent the real structure from multiple modes with inconsistent possible directions, laying the foundation for subsequent projection fusion.

[0093] Furthermore, step F2 projects the gradient of each mode onto the globally dominant direction e through a projection operation. max On (x), the projected gradient is obtained: The gradients of all modes are unified into the same reference direction system, which makes the subsequent weighted average have a clear geometric meaning and fundamentally avoids the mutual cancellation problem caused by inconsistent gradient directions.

[0094] Step F3 employs a weighted fusion strategy similar to that in Example 1, but the fusion target is changed from the original gradient. I i (x) becomes the projected gradient in, The fusion weights are still based on the degree of degradation and the gradient magnitude. If needed, the enhancement factor g(x) from Example 1 can still be applied subsequently. The fused gradient field obtained after step F3 has a gradient direction at each pixel that aligns with the most reliable structural direction in that local region, while the gradient magnitude is dominated by the mode with less degradation and stronger structure. This gradient field is used for subsequent weighted least squares reconstruction, which can generate enhanced fused images with sharp edges, clear structure, and no ghosting.

[0095] Furthermore, step F2 above calculates the gradients of all modes. I i The "hard projection" strategy of projecting (x) onto the globally dominant direction is effective in edge regions (with a clear dominant direction), but it introduces new problems in the following scenarios: Flat regions: such as the sky, walls, and other areas with uniform grayscale in an image, have very small gradient magnitudes, and the two eigenvalues ​​of the structure tensor are similar. The dominant direction is randomly determined by tiny noise. In this case, forced projection will "legitimize" the direction of random noise, resulting in false directionality in the fused gradient field, which in turn causes unnatural textures in the reconstructed image.

[0096] Isotropic texture regions: such as grass, sand, and other areas with messy textures, where gradients are evenly distributed in all directions without a clear dominant direction. Forced projection will lose a lot of texture information perpendicular to the "pseudo-dominant direction," resulting in blurred or lost texture details.

[0097] Multi-directional intersection areas, such as corners and intersections, have multiple significant directions. A single dominant direction cannot fully describe the local structure, and forced projection will lose information about secondary directions, resulting in structural distortion.

[0098] To this end, this embodiment introduces a local structure metric—coherence—to distinguish between different structure types and to adopt the most suitable gradient processing strategy for each type.

[0099] Step C, which involves calculating the fused gradient field G(x), also includes a step of adaptive structural projection, replacing step F2: Step G1: For the weighted average structure tensor Perform eigenvalue decomposition and calculate the local coherence Coh(x) based on the eigenvalues ​​obtained from the decomposition. Step G2: Divide the pixels into different types of regions according to the coherence Coh(x), and apply a strategy that matches the different types of regions to the modal gradient. I i (x) is processed to obtain the processed gradient. I i type (x); Step G3: Based on the degradation level map D i (x) and gradient magnitude | I i (x)∣, for the processed gradient I i type (x) is weighted and fused.

[0100] For step G1: based on the weighted average structure tensor already calculated in step F1 And its eigenvalues, defining coherence: Where η is a very small positive constant used to prevent division by zero, λ1 represents the gradient strength along the dominant direction, and λ2 represents the gradient strength perpendicular to the dominant direction.

[0101] When λ1 < λ2 (typically in the edge region), Coh(x) ≈ 1, indicating that the local structure is highly anisotropic and there is a clear dominant direction.

[0102] When λ1≈λ2 and both are large (in areas with messy textures), Coh(x)≈0, indicating that the local structure is isotropic and the gradient is uniformly distributed in all directions.

[0103] When λ1≈λ2≈0 (flat region), Coh(x)≈0, but the total gradient intensity is very low at this time, and it also exhibits isotropy.

[0104] By using coherence, a simple yet effective metric, we can accurately distinguish different types of local structures in an image, providing a basis for decision-making in subsequent adaptive processing.

[0105] For step G2: Set two thresholds T high and T low (usually 0) <T low <T high <1), the pixels are divided into three regions, and different strategies are used for each modal gradient. I i (x) is processed to obtain the processed gradient. I i type (x).

[0106] This embodiment designs the most suitable gradient processing strategy for different types of regions. The regions mentioned in step G2 include high coherence regions, low coherence regions, and medium coherence regions. For the high coherence region (Coh(x)≥T) high The processing strategy is to project the gradient onto the globally dominant direction e. max On (x), the projected gradient is obtained; Applicable scenarios: Typical edge areas with a clear dominant direction. Hard projection can eliminate directional conflicts and enhance edge consistency.

[0107] The principle is as follows: In such areas, structural information is mainly concentrated in the dominant direction. The projection operation retains the main information, and the discarded vertical components usually correspond to noise or minor texture undulations, which do not affect the main structure.

[0108] For the low coherence region (Coh(x)≤T) low The processing strategy is to preserve the original gradient. Ii (x); Applicable scenarios: flat regions (low total gradient strength) and textured regions (isotropic). Neither of these types of regions has a reliable dominant direction.

[0109] The principle is as follows: In flat regions, gradients are primarily noise, and any projection operation will organize the random noise into a false "directional texture." Preserving the original gradients and allowing them to cancel each other out in the subsequent weighted averaging actually has a smoothing effect. In regions with cluttered textures, texture information in all directions is equally important. Hard projection loses texture perpendicular to the "pseudo-dominant direction," while preserving the original gradient completely retains isotropic texture features.

[0110] For the moderate coherence region (T) low ≤Coh(x)≤T high The processing strategy is soft projection, which simultaneously retains the dominant directional component and a portion of the vertical directional component to obtain the soft-projected gradient image. I i soft (x).

[0111] Among them, e min (x) is perpendicular to e max The direction of (x) (i.e., the eigenvector corresponding to the minimum eigenvalue of the structure tensor), μ∈[0,1] is the coefficient for retaining the vertical component.

[0112] Suitable scenarios: Transition areas between edges and textures, or edge areas containing secondary structures (such as feathers, hair, etc.). The principle is as follows: In such regions, there is both a dominant directionality and non-negligible vertical direction information. Hard projection will lose this secondary information, while isotropic fusion may introduce directional conflicts.

[0113] Soft projection controls the degree of vertical component preservation through the parameter μ, achieving a smooth transition from hard projection to isotropic fusion. μ can be set as a function related to coherence, such as μ=1. Coh(x) makes the higher the coherence, the stronger the vertical component suppression.

[0114] This embodiment achieves the ideal effect of "preserving direction in edge areas, smoothness in flat areas, detail in texture areas, and continuity in transition areas" through region classification and adaptive processing, enabling the fused gradient field to more realistically reflect the local structural characteristics of the image.

[0115] Step G3 employs the same weighted fusion strategy as in Example 1, but the object of fusion is changed to the adaptively processed gradient, i.e.: The fused gradient field obtained after step G3 is not only accurate in direction and reasonable in magnitude, but more importantly, it highly matches the local structure type of the image—sharp and consistent at edges, smooth and clean in flat areas, and rich in detail in textured areas. Such a gradient field is used for subsequent weighted least squares reconstruction, which can generate enhanced fused images with higher visual quality and more natural appearance.

[0116] In summary, based on Example 1, this embodiment uses adaptive processing to fuse gradient fields, enabling the most suitable strategies to be adopted in different regions, thereby achieving the best visual effect in each region of the final image.

[0117] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for image enhancement and restoration in harsh environments based on multimodal feature fusion, characterized in that, Includes the following steps: Step A: Acquire N modal images of the same scene, as the multimodal image I to be processed. i Where i represents the corresponding modal image change, i = 1, 2, ..., N, N ≥ 2; Step B: Based on the pre-trained first neural network model DegNet, obtain and calculate the I of each modality image. i Degradation level diagram D i (x); where the first neural network model DegNet takes a multimodal image patch as input and outputs a corresponding degradation level map D. i (x); the degradation level diagram D i (x) is used to characterize the degree of degradation at each pixel position x in the image; Step C: Based on the degradation level map D i (x) and the gradient field of each modality image I i (x), calculate the fusion gradient field G(x); Step D: Construct an energy function E(F) for the enhanced fused image F to be solved, the energy function including a pixel fidelity term and a gradient fidelity term; wherein, the pixel fidelity term constrains the enhanced fused image F and the reference image I. ref The gradient fidelity term constrains the difference between the gradient field of the enhanced fused image F and the fused gradient field G(x); Step E: The enhanced fused image F is obtained by minimizing the energy function E(F).

2. The method for image enhancement and restoration in harsh environments based on multimodal feature fusion according to claim 1, characterized in that, The first neural network model, DegNet, uses different prior methods to obtain degradation maps for training; wherein, the prior methods include: Degradation of visible light images in foggy weather is estimated based on transmittance estimation using dark channel priors. The degree of degradation of infrared images is estimated based on the noise level estimation using local standard deviation. The degradation level of low-light images is estimated based on the mean available brightness.

3. The method for image enhancement and restoration in harsh environments based on multimodal feature fusion according to claim 1, characterized in that, The calculation of the fused gradient field G(x) in step C specifically includes: Step C1: According to the degradation degree diagram D i (x) and gradient magnitude | I i (x)∣, calculate the fusion weights for each modality. Among them, the higher the degree of degradation, the smaller the weight; Step C2: Gradient field for each mode I i (x) is weighted and averaged to obtain the weighted average gradient. ; Step C3: Based on the degradation degree of the local clearest mode min i D i (x), calculate the enhancement factor g(x), which is negatively correlated with the degree of degradation of the clearest mode; Step C4: Apply the enhancement factor g(x) to the weighted average gradient. I avg (x), thus obtaining the fused gradient field G(x) = g(x). I avg (x).

4. The method for image enhancement and restoration in harsh environments based on multimodal feature fusion according to claim 3, characterized in that, The fusion weights described in step C1 The calculation formula is: Where α is a weight control parameter greater than 0, ε and δ are constants to prevent division by zero, and j is the summation iteration variable.

5. The method for image enhancement and restoration in harsh environments based on multimodal feature fusion according to claim 3, characterized in that, The formula for calculating the enhancement factor g(x) in step C3 is as follows: Where β is the enhancement amplitude control parameter greater than 0, and γ is the decay rate control parameter greater than 0.

6. The method for image enhancement and restoration in harsh environments based on multimodal feature fusion according to claim 1, characterized in that, The specific form of the energy function E(F) mentioned in step D is: Where p is the pixel index. G represents the neighborhood of pixel p. pq To fuse the projection of the gradient field G(x) onto the direction from pixel p to its neighboring pixel q, λ is a parameter balancing the pixel fidelity term and the gradient fidelity term; this is achieved by solving the corresponding linear equation system (I+λD). D)F=I ref +λD G yields the enhanced fused image F, where I is the identity matrix and D is the discrete difference matrix.

7. The method for image enhancement and restoration in harsh environments based on multimodal feature fusion according to claim 1, characterized in that, Step C, which involves calculating the fused gradient field G(x), also includes a step of constraining the direction consistency. Step F1: Based on the gradient field of each modality image I i (x), calculate the local structure tensor and obtain the global dominant direction e. max (x); Step F2: Convert the gradients of each mode I i (x) Projected onto the global dominant direction e max On (x), the projected gradient is obtained; Step F3: Based on the degradation level map D i (x) and gradient magnitude | I i (x)∣, weighted fusion of the projected gradients yields a fused gradient field with consistent direction.

8. The method for image enhancement and restoration in harsh environments based on multimodal feature fusion according to claim 7, characterized in that, The global dominant direction e mentioned in step F1 max (x) is obtained by the following method: for the structure tensor S of each mode. i (x) is weighted and averaged to obtain the weighted average structure tensor. The weights are the fusion weights. ; for the weighted average structure tensor Perform eigenvalue decomposition and take the eigenvector corresponding to the largest eigenvalue as the global dominant direction e. max (x).

9. A method for image enhancement and restoration in harsh environments based on multimodal feature fusion according to claim 8, characterized in that, Step C, which involves calculating the fused gradient field G(x), also includes a step of adaptive structural projection, replacing step F2: Step G1: For the weighted average structure tensor Perform eigenvalue decomposition and calculate the local coherence Coh(x) based on the eigenvalues ​​obtained from the decomposition. Step G2: Divide the pixels into different types of regions according to the coherence Coh(x), and apply a strategy that matches the different types of regions to the modal gradient. I i (x) is processed to obtain the processed gradient. I i type (x); Step G3: Based on the degradation level map D i (x) and gradient magnitude | I i (x)∣, for the processed gradient I i type (x) is weighted and fused.

10. A method for image enhancement and restoration in harsh environments based on multimodal feature fusion according to claim 9, characterized in that, The region types mentioned in step G2 include high coherence regions, low coherence regions, and medium coherence regions; For regions of high coherence, the processing strategy is to project the gradient onto the globally dominant direction e. max On (x), the projected gradient is obtained; For regions with low coherence, the processing strategy is to preserve the original gradient. I i (x); For regions of moderate coherence, the processing strategy is soft projection, which simultaneously preserves the dominant directional component and a portion of the vertical directional component to obtain the soft-projected gradient image. I i soft (x).

Images