Infrared and visible multiscale fusion system based on deep feature extraction

Abstract

The application relates to the technical field of image processing, and discloses an infrared and visible light multiscale fusion system based on deep feature extraction, which comprises a data preprocessing module, a feature mapping module, a feature decoupling module and an image reconstruction module. The system establishes geometric mapping of infrared and visible light images, extracts deep features, maps the features to a unified feature vector space, removes redundant components linearly related to infrared features in visible light features based on the vector orthogonalization principle, suppresses transient light interference in time sequence by using thermal inertia logic, extracts pure texture features, constructs a guide vector field perpendicular to an infrared temperature gradient, and constrains anisotropic diffusion of the texture features along the isotherm direction and fusion. The application can effectively eliminate redundant information between modes, suppress dynamic light noise while maintaining the edge integrity of an infrared thermal target, and improve the definition and target recognition degree of a fused image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to an infrared-visible multi-scale fusion system based on depth feature extraction. Background Technology

[0002] Infrared thermal imaging technology can image based on differences in thermal radiation on the surface of objects, has all-weather operation and the ability to penetrate smoke, and can highlight heat source targets. However, its imaging lacks texture details, has low contrast, and is difficult to reflect the geometric structure of a scene. Visible light images contain rich color and texture information, conforming to human visual habits, but are easily affected by lighting conditions, weather conditions, and obstructions. Fusion of infrared and visible light images can combine the advantages of both to generate a fused image that is both salient to thermal targets and contains clear texture details, which has application value in fields such as security monitoring, assisted driving, and military reconnaissance.

[0003] Existing infrared and visible light image fusion methods mainly include traditional methods based on multi-scale transformations and deep learning-based methods. With the development of deep neural networks, fusion algorithms based on convolutional neural networks or autoencoders have gradually become mainstream. However, existing deep learning fusion frameworks typically focus on increasing the number of network layers or designing complex attention mechanisms, often employing channel concatenation or simple element-wise weighting strategies at the feature processing level. This approach ignores the correlation between different modal signals, leading to the repeated superposition of edge contour information shared by infrared and visible light images. This easily results in edge ghosting or blurring in the fused image, reducing the overall image clarity.

[0004] Furthermore, visible light images are extremely sensitive to changes in ambient lighting and are easily affected by non-object textures such as reflections, glare, or shadows. Most existing fusion algorithms lack consideration for the dynamic characteristics of the physical scene, making it difficult to effectively distinguish between real object textures and transient lighting noise. When strong light interference occurs in the scene, existing fusion systems often mistake lighting noise for high-frequency texture features and inject it into the fused image, thereby masking the true infrared thermal target and affecting the accurate identification of the target.

[0005] In the image reconstruction stage, existing methods typically employ isotropic feature injection, which does not fully consider the geometric distribution characteristics of the infrared thermal radiation field. When visible light textures are directly superimposed on infrared features, without directional constraints, the texture features can easily cross the boundaries of the thermal target, disrupting the original temperature gradient distribution of the infrared image. This results in blurred outlines of the thermal target, weakening the advantages of infrared images in target detection. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an infrared and visible light multi-scale fusion system based on depth feature extraction, which solves the problems in existing infrared and visible light image fusion technologies, such as image blurring due to the superposition of redundant information between modes, susceptibility to dynamic lighting noise interference, and texture injection that destroys the edge contours of infrared thermal targets.

[0007] To achieve the above objectives, the present invention provides the following technical solution: an infrared-visible multi-scale fusion system based on depth feature extraction, comprising:

[0008] The data preprocessing module is used to establish the geometric mapping relationship between infrared images and visible light images, perform pixel-level spatial registration and signal standardization processing, and generate input tensors in a unified format that are strictly aligned in the spatiotemporal dimension.

[0009] The feature mapping module, connected to the data preprocessing module, uses a dual-stream network architecture to extract the deep semantic features of infrared and visible light images respectively, and maps the heterogeneous features to a unified feature vector space through channel projection operation.

[0010] The feature decoupling module, connected to the feature mapping module, is used to perform feature orthogonalization decomposition in the feature vector space, separate and remove parallel components that are linearly related to infrared features in the visible light features, retain orthogonal texture components, detect the temporal dynamic changes of the texture components, suppress transient interference based on physical thermal inertia logic, and output purified visible light features.

[0011] The image reconstruction module, connected to the feature mapping module and the feature decoupling module, is used to analyze the spatial gradient distribution of infrared features, construct a guiding vector field perpendicular to the temperature gradient direction, constrain the purified visible light features in the infrared features to undergo anisotropic diffusion fusion only along the direction of the guiding vector field, and reconstruct a multi-scale fused image.

[0012] The data preprocessing module includes a parameter calibration unit and a spatial registration unit. The parameter calibration unit acquires the intrinsic parameter matrices, distortion coefficients, rotation matrix, and translation vector between the infrared thermal imager and the visible light camera. The spatial registration unit constructs a homography matrix based on the intrinsic parameter matrix, rotation matrix, translation vector, and reference plane normal vector. It then uses the distortion coefficients to perform distortion correction on the original image and performs perspective transformation interpolation on the distortion-corrected visible light image based on the homography matrix to make its field of view coincide with that of the infrared image.

[0013] In one specific implementation, the data preprocessing module further includes a data tensor unit. For infrared images, their radiation intensity data is linearly mapped to a normalized range as the infrared input tensor; for visible light images, they are converted from the RGB color space to the YCbCr color space, and the luminance channel data is extracted as the visible light input tensor.

[0014] Regarding the network structure for feature extraction, the dual-stream neural network in the feature mapping module includes infrared feature extraction and visible light feature extraction branches. The two branches maintain symmetry in the network structure but do not share weight parameters during training, thus learning thermal radiation feature representation and reflected light texture feature representation respectively. Furthermore, a dimension projection alignment unit is included, connecting convolutional layers with a kernel size of 1 to the outputs of both branches. Infrared and visible light features are projected to the same channel dimension through a linearly weighted combination of channel dimensions. A batch normalization layer is connected after the convolutional layers instead of a non-linear activation function to preserve the directional information of the feature vectors, providing a mathematical basis for subsequent orthogonal decomposition.

[0015] Regarding the specific implementation of feature decoupling, the feature decoupling module includes a vector orthogonal projection unit. The vector orthogonal projection unit treats pixels on the feature map as feature vectors, calculates the projection vector of the visible light feature vector onto the direction of the infrared feature vector as a parallel component, and calculates the difference between the visible light feature vector and the parallel component through vector subtraction to obtain the orthogonal component. The orthogonal component characterizes the texture structure information in the visible light image that is independent of the infrared thermal radiation distribution.

[0016] Furthermore, to enhance the highlighting of thermal targets, the feature decoupling module also includes a thermal saliency calculation unit. This unit calculates the mean and variance of the local neighborhood features for each pixel in the infrared image, performs a weighted summation of the two, and maps the summation result through a Sigmoid activation function to generate thermal saliency weights. Based on these thermal saliency weights, orthogonal components are weighted and suppressed, thereby reducing background texture interference in insignificant areas.

[0017] To address interference suppression in dynamic scenes, the feature decoupling module further includes a temporal alignment unit and a thermal inertia gating unit. The temporal alignment unit uses an optical flow algorithm to calculate the optical flow field between the current and previous moments and aligns the feature data from the previous moment to the coordinate system of the current moment. The thermal inertia gating unit calculates the magnitude of the difference between the orthogonal components of the current moment and the aligned orthogonal components of the previous moment as the rate of change of time. When the rate of change of time is greater than a preset thermal inertia threshold, the corresponding region is determined to be an illumination interference region and a suppression mask is generated; otherwise, it is determined to be a valid texture region conforming to physical thermal inertia.

[0018] In the image reconstruction stage, the image reconstruction module includes a gradient field calculation unit and an anisotropic diffusion fusion unit. The gradient field calculation unit uses an edge detection operator to calculate the horizontal and vertical gradients of the infrared features, constructing a gradient vector field. The anisotropic diffusion fusion unit performs orthogonal rotation and normalization on the gradient vectors in the gradient vector field to construct a guiding vector field, wherein the guiding vector field is an isotherm tangential vector field indicating the direction in which the temperature value remains unchanged in a local region.

[0019] Specifically, the anisotropic diffusion fusion unit constructs a diffusion tensor, which is a linear combination of the outer product of the tangential vectors of the isotherm tangential vector field and the outer product of the gradient vectors of the gradient vector field. The weighting coefficients corresponding to the outer product of the tangential vectors are set to be greater than those corresponding to the outer product of the gradient vectors, so that the purified visible light features diffuse primarily along the isotherm direction. Finally, the decoding and reconstruction unit superimposes the infrared features with the anisotropically diffused visible light features and reconstructs a fused image through a decoding network including an upsampling layer.

[0020] This invention provides an infrared-visible multi-scale fusion system based on depth feature extraction. It has the following beneficial effects:

[0021] 1. This invention separates visible light features into parallel components linearly related to infrared features and independent perpendicular components by performing orthogonal decomposition in the feature vector space, and then removes the parallel components. This processing method mathematically removes redundant contour information from infrared and visible light images, avoiding edge ghosting or blurring caused by direct fusion, ensuring that the final fused image retains only the unique texture details of visible light, and improving image contrast and visual recognition.

[0022] 2. This invention distinguishes between valid texture and illumination interference by calculating the rate of change of feature components over consecutive frames. Since the thermal radiation distribution of physical targets has thermal inertia and does not undergo drastic changes in a short period, the system can identify and filter out transient high-frequency signals caused by reflections, glare, etc. This allows the system to accurately extract realistic texture features even in dynamic scenes with complex or changing lighting conditions, reducing the introduction of false information.

[0023] 3. This invention employs an anisotropic diffusion strategy constrained by the isotherm tangential vector field, forcing visible light texture features to diffuse and fuse along the isotherm direction of the infrared image, rather than crossing edges along the gradient direction. This mechanism ensures that the injection of texture information does not destroy the original contour boundaries of the infrared thermal target, preventing temperature gradient blurring caused by texture coverage. Thus, while enriching the internal details of the target, it fully preserves the detection advantages of infrared images for thermal targets. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the system framework of the present invention;

[0025] Figure 2 This is a schematic diagram illustrating the operational logic of the data preprocessing module of the present invention;

[0026] Figure 3 This is a schematic diagram illustrating the operational logic of the feature mapping module of the present invention;

[0027] Figure 4 This is a schematic diagram of the operational logic of the feature decoupling module of the present invention;

[0028] Figure 5 This is a schematic diagram illustrating the operational logic of the image reconstruction module of the present invention;

[0029] Figure 6 This is a quantitative comparison chart of the performance indicators of different fusion algorithms of the present invention.

[0030] Among them, 100 is the data preprocessing module; 200 is the feature mapping module; 300 is the feature decoupling module; and 400 is the image reconstruction module. Detailed Implementation

[0031] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Please see the appendix Figure 1 This invention provides an infrared-visible multi-scale fusion system based on depth feature extraction, comprising:

[0033] The data preprocessing module 100 is configured to acquire infrared and visible light images of the target under test, perform binocular correction and spatial registration, eliminate parallax, output infrared and visible light image data aligned in the pixel coordinate system, and convert the image data into input tensors in a unified format.

[0034] The feature mapping module 200 is connected to the data preprocessing module 100 and is configured to extract the depth features of the infrared image and the visible light image respectively using a two-stream neural network, and map the infrared feature tensor and the visible light feature tensor to the feature vector space of the same dimension through channel projection operation.

[0035] The feature decoupling module 300 is connected to the feature mapping module 200 and is configured to calculate the thermal saliency weight of infrared features. Based on the principle of vector orthogonalization, the visible light features are decomposed into components parallel to the infrared features and orthogonal components perpendicular to the infrared features. The parallel components are removed, the orthogonal components are retained, and the time change rate of the orthogonal components between consecutive frames is calculated. When the time change rate exceeds the preset thermal inertia threshold, the orthogonal components are suppressed, and the purified visible light features are output.

[0036] The image reconstruction module 400 is connected to the feature mapping module 200 and the feature decoupling module 300 respectively. It is configured to calculate the spatial gradient field of infrared features and the isotherm tangential vector field perpendicular to the gradient. Based on the constraint of the tangential vector field, the purified visible light features diffuse along the isotherm direction and are fused into the infrared features. The fused feature tensor is reconstructed into a fused image through the decoding network.

[0037] The specific implementation principles of each module of the present invention will be explained in detail below with reference to the accompanying drawings.

[0038] The data preprocessing module 100, serving as the system's input front-end, primarily undertakes the tasks of geometric alignment and signal normalization of multimodal data. In this embodiment, the data preprocessing module is physically composed of an image acquisition interface connected via a data bus and a digital signal processing unit (such as an FPGA pipeline or GPU core). To ensure a strict spatial correspondence between infrared thermal radiation data and visible light images, and to adapt the numerical distribution to subsequent neural networks, the data preprocessing module 100 specifically performs sensor calibration, spatial registration, and signal tensor quantization processing.

[0039] See attached document Figure 2 The data preprocessing module 100 specifically includes a parameter calibration unit, a spatial registration unit, and a data tensor quantization unit. In this embodiment, the data preprocessing module 100 serves as the input front-end of the system, converting the original signals, which exhibit disparity and spectral response differences in physical space, into standard tensors that are geometrically aligned and whose data distribution is suitable for convolutional neural network processing.

[0040] In this embodiment, the parameter calibration unit establishes a geometric mapping model between the infrared thermal imager and the visible light camera. Since the infrared sensor and the visible light sensor have a baseline distance in their physical locations, and their lenses have different optical characteristics, it is necessary to obtain their respective intrinsic parameter matrices, distortion coefficients, and relative pose relationships. Using a pre-set calibration target (such as a checkerboard or dot array made of different materials), the parameter calibration unit calculates the intrinsic parameter matrix of the infrared camera. Intrinsic parameter matrix of a visible light camera Radial distortion coefficient of infrared lens With tangential distortion coefficient And the rotation matrix of the infrared camera coordinate system relative to the visible light camera coordinate system. Translation vector For the specific calibration algorithm solution process, those skilled in the art can use the Zhang Zhengyou calibration method or other conventional optoelectronic system calibration techniques. The specific iterative solution process is a well-known technique in this field and will not be elaborated here.

[0041] Based on the aforementioned calibration parameters, the spatial registration unit performs pixel-level geometric correction. Before performing coordinate mapping, the spatial registration unit first utilizes the distortion coefficients... The original acquired image undergoes distortion correction processing to restore the nonlinear optical path to an ideal linear pinhole model, thereby eliminating geometric distortion at the lens edges. This process is applied to the distortion-corrected infrared image. coordinates of any pixel in The spatial registration unit 120 calculates its representation in the visible light image based on the geometric projection relationship. Corresponding coordinates in Considering that in industrial monitoring scenarios, the devices under test are typically located at a specific monitoring distance. In this case, or assuming the monitored scene is approximately planar, the system uses a perspective transformation matrix for image registration. The coordinate mapping relationship is expressed as follows:

[0042] ;

[0043] In the formula, This represents the homogeneous coordinate scale factor, used for normalization. Represents the range from infrared to visible light. Homography matrix, These represent the horizontal and vertical coordinates of a pixel in an infrared image, respectively. These represent the horizontal and vertical coordinates of the corresponding pixels in the visible light image, respectively. , These represent the infrared and visible light imaging channels, respectively, and 1 represents the homogeneous coordinate component.

[0044] In this embodiment, the homography matrix From the intrinsic parameter matrix , extrinsic rotation matrix Translation vector and the reference plane normal vector and distance They are jointly determined, and their computational relationship satisfies The spatial registration unit performs inverse interpolation transformation (such as bilinear interpolation) on the visible light image based on the calculated coordinate mapping relationship, generating a corrected visible light image with the same resolution and field of view as the infrared image, and outputting the infrared image. With visible light images Strict spatial alignment is achieved in the pixel coordinate system, meaning that the same coordinate positions in the image are aligned. The same physical point on the surface of the object being measured.

[0045] The data tensor quantization unit is used to convert aligned image data into a normalized tensor format required by deep learning networks. This is specifically for infrared images. The raw data is typically 14-bit or 16-bit high dynamic range data reflecting radiation intensity. The data tensor unit performs a linear normalization operation, mapping the radiation intensity to the [0,1] interval to fit the input range of the neural network activation function. The normalization calculation formula is:

[0046] ;

[0047] in, This indicates the aligned infrared image in pixel coordinates. The original radiation intensity value at that location, This indicates that the normalized infrared image pixel values ​​range from [0, 1]. Represents the pixel coordinates of the image. This represents the normalized infrared data. and The value is determined using either a fixed range strategy or a dynamic statistical strategy: in quantitative temperature measurement mode, and Set as the boundary value of the physical temperature measurement range of the infrared thermal imager (e.g., -20℃ to 150℃) to maintain the absolute physical meaning of the temperature values; in enhanced display mode, and The minimum and maximum values ​​of pixel grayscale in the current frame image are set to maximize local contrast. The normalized data is then expanded into a single-channel tensor.

[0048] For visible light images The data tensor unit converts it from the RGB color space to the YCbCr color space and separates the luminance channel. The principle behind this process is that in multi-scale fusion, visible light texture details are primarily carried in the luminance component, while the chrominance component has a weaker correlation with infrared thermal features. Using only the luminance channel as network input reduces computational redundancy and allows focus on fusing structural information. The color space conversion formula is as follows:

[0049] ;

[0050] In the formula, Represents the visible light image in pixel coordinates The red channel pixel value at that location, Represents the visible light image in pixel coordinates The green channel pixel value at that location, Represents the visual image in pixel coordinates The blue pixel value This represents the brightness obtained from RGB conversion. , These represent the pixel coordinates of the image.

[0051] The luminance channel is extracted and calculated by the data tensor quantization unit. It is constructed as a single-channel tensor as the visible light input branch of the network, and the data preprocessing module 100 outputs two dimensions. The standard tensor.

[0052] See attached document Figure 3 The feature mapping module 200 specifically includes a two-stream feature extraction unit and a dimension projection alignment unit. In this embodiment, the feature mapping module 200 is connected to the data preprocessing module 100, and aims to extract high-dimensional semantic features from heterogeneous source images and map feature signals with different physical properties to a unified mathematical manifold space, so as to facilitate subsequent modules to perform vector-based algebraic operations.

[0053] In this embodiment, the dual-stream feature extraction unit is configured as an independent dual-channel network topology, corresponding to the infrared feature extraction branch and the visible light feature extraction branch, respectively. Although the two branches maintain structural symmetry in terms of network layers and connection methods (e.g., both using the front-end layer of ResNet or a VGG-type stacked structure), they do not share weights in the parameter space. The lack of weight sharing refers to the convolution kernel parameters of the infrared branch. Convolution kernel parameters with visible light branch During training, updates are independent and unconstrained. This follows the principle of feature domain separation, ensuring the network can learn smooth feature representations adapted to thermal radiation characteristics and gradient feature representations adapted to reflected light characteristics, thus avoiding semantic confusion caused by single-weight networks when processing heterogeneous data.

[0054] Specifically, the infrared branch receives the normalized infrared tensor, and performs nonlinear mapping through consecutively stacked 3×3 convolutional layers, batch normalization layers, and nonlinear activation functions to generate the original infrared feature tensor. The visible light branch processes the visible light luminance tensor using the same hierarchical structure to generate the original visible light feature tensor. To ensure that the spatial resolution of the output feature map remains consistent with that of the input image (i.e. The two-stream feature extraction unit eliminates the max-pooling or average-pooling layers found in conventional convolutional neural networks and sets the stride of all convolutional layers to 1. For cases requiring a larger receptive field to capture global semantics, the two-stream feature extraction unit replaces downsampling operations with dilated convolutions with a dilation rate greater than 1.

[0055] The dimensionality projection alignment unit is located at the output end of the feature extraction network to address the issues of potentially inconsistent original feature channel numbers and misalignment of the feature space basis. Since the infrared and visible light branches are optimized separately, their output feature channel numbers may differ, and the distribution range of feature values ​​may vary. The dimensionality projection alignment unit introduces a [missing information - likely a specific feature] at the end of each branch. Convolutional layer. Convolutional layers force infrared and visible light features to be mapped to the same preset channel dimension through linear weighted combination along the channel dimension. (For example =64).

[0056] In particular, to ensure the mathematical validity of subsequent vector orthogonality operations, Instead of using nonlinear activation functions like ReLU after the projective convolutional layer, a batch normalization layer is directly connected. The physical principle behind this design is that linear outputs allow feature vectors to take negative values ​​in the feature space, thus preserving the vector's directional information; while the batch normalization layer constrains the feature distribution to near zero mean and unit variance, preventing numerical instability during projection calculations due to excessive differences in feature amplitudes. After the projection transformation, the output infrared feature tensor... With visible light feature tensor They are completely identical in mathematical dimension and both belong to... The calculation process for spatial dimension projection alignment units can be represented as follows:

[0057] ;

[0058] ;

[0059] In the formula, This indicates a batch normalization operation. This represents a convolution operation with a kernel size of 1. Represents the original feature tensor of the infrared modes. This represents the original feature tensor of the visible light mode. This operation not only achieves physical alignment of dimensions, but more importantly, through learnable projection parameters, it implicitly transforms two heterogeneous signals into the same metric space, making the subsequent calculation of the dot product or distance between the two feature vectors have practical physical and mathematical meaning.

[0060] To support subsequent feature orthogonal decoupling operations, this embodiment explicitly defines the data structure on the feature map in a vectorized manner. For any pixel coordinate position in the feature tensor space... It is no longer considered a single value, but is defined as a A dimensional eigenvector. The specific definition is as follows:

[0061] ;

[0062] ;

[0063] In the formula, Represents the spatial pixel coordinates in the feature map. The alignment feature tensor representing the infrared and visible light modes. Indicates position Infrared feature vector at that location, Indicates position Visible light feature vector at that location, This represents the channel dimension of the feature vector. express A real vector space.

[0064] Based on this definition, the feature fusion problem across the entire image plane is transformed into... An independent vector space geometric projection problem.

[0065] See attached document Figure 4 The feature decoupling module 300 specifically includes a thermal saliency calculation unit, a vector orthogonal projection unit, a temporal alignment unit, and a thermal inertia gating unit. In this embodiment, the feature decoupling module 300, as the core processing unit of the present invention, aims to solve the technical problem of visible light textures destroying quantitative features of infrared thermal radiation from both mathematical and physical perspectives. The feature decoupling module receives infrared feature tensors from the feature mapping module 200. With visible light feature tensor It outputs purified visible light characteristics. .

[0066] In this embodiment, the thermal saliency calculation unit is configured to quantify the importance of each pixel in the infrared image for thermal fault diagnosis and generate a thermal saliency weight map. Considering that in infrared thermal imaging, high-radiation-intensity regions (high-temperature points) and high-radiation-gradient regions (edges) often contain crucial fault information, the thermal saliency calculation unit employs a hybrid metric method based on local statistical features. For any pixel location... thermal significance weight The calculation formula is as follows:

[0067] ;

[0068] In the formula, The Sigmoid activation function is used to map the output to the (0,1) interval. and They represent respectively with A local neighborhood window centered on the center (e.g.) Mean and variance of infrared characteristic amplitudes within ) This is a normalization operator used to linearly scale input data to the range [0,1]. and These are the preset weighting coefficients (e.g.) This is important for balancing absolute temperature and temperature gradient.

[0069] thermal significance weighting plot It intuitively reflects whether the current area belongs to the thermal sensitive area. The closer the value is to 1, the less the thermal information of the area can be covered.

[0070] The vector orthogonal projection unit, based on the Gram-Schmidt orthogonalization principle in linear algebra, decouples visible light features in the eigenvector space RCRC. The physical principle of this method lies in the fact that visible light feature vectors... Mid-infrared feature vector Parallel components physically represent structural information shared by the two modes (such as the geometric contours of an object). Directly superimposing this information can lead to ghosting at image edges or abnormal contrast. Components perpendicular to the infrared features, on the other hand, represent unique texture information not present in the infrared image (such as printed characters or color boundaries). Therefore, in this embodiment, the vector orthogonal projection unit first calculates the visible light feature vector. In infrared feature vectors Parallel projection components in the direction ;

[0071] ;

[0072] In the formula, Indicates position Visible light feature vector at that location, Indicates position Infrared feature vector at that location, express exist Projection in direction, Represents the L2 norm. To prevent small constants with a denominator of zero (e.g., 10) -6 Subsequently, the vector orthogonal projection unit extracts the orthogonal components by stripping the parallel components through vector subtraction. :

[0073] ;

[0074] In the formula, express relatively The orthogonal components.

[0075] Orthogonal components This constitutes the initial texture feature tensor. It mathematically guarantees that it is uncorrelated with infrared thermal features, that is, adding texture will not change the projection amplitude of infrared features in the original direction, thereby maximizing the protection of the quantitative accuracy of thermal data.

[0076] The timing alignment unit and thermal inertia gating unit introduce a time dimension, using physical thermal inertia to further filter out false textures (such as specular reflections on metal surfaces or dynamic lighting interference). Because the thermal changes of industrial equipment have significant thermal inertia, the rate of change of its temperature field is limited by the specific heat capacity and thermal conductivity of the material, and is usually slow; while the changes in light and shadow under visible light (such as strobe lights and shadows of people walking) are often transient.

[0077] In this embodiment, the timing alignment unit first obtains the previous time step. orthogonal characteristic tensor With the current moment Feature tensor To eliminate misjudgments caused by minute camera shakes or movements, the temporal alignment unit uses optical flow algorithms (such as the Lucas-Kanade algorithm or the FlowNet network) to calculate the dense optical flow field between two frames. and will The features at each time step are aligned to the coordinate system of the current time step to obtain the corrected features. The specific implementation of optical flow alignment is a well-known technique in the field of computer vision, and will not be elaborated here.

[0078] The temporal rate of change of characteristic intensity calculated by the thermal inertial gating unit A binary mask is constructed based on the physical thermal inertia threshold. The rate of change is defined as the magnitude of the difference between the aligned eigenvectors:

[0079] ;

[0080] In the formula, Indicates position The temporal rate of change of the characteristic intensity at that location, Represents the spatial coordinates of the feature map. Indicates time orthogonal characteristics, Indicates time After the features are aligned / deformed, The value at that location, Indicates the index of the current frame / current time. Indicates the index of the previous frame / previous time. This indicates the time interval between frames.

[0081] Subsequently, the thermal inertia gating unit compares the rate of change with a preset thermal inertia threshold. Comparison. In order to determine. For the specific value, this embodiment adopts the statistical calibration method: during the system initialization phase, a static background data with stable illumination is collected, and the mean value of the characteristic change rate during this period is calculated. with standard deviation ,set up (in (This is a confidence coefficient, for example, 3). This thermal inertia threshold represents the upper limit of the characteristic change rate caused by sensor noise and small environmental fluctuations under conditions of no external light interference. If the change in the detection area violates the physical law of thermal inertia, it is considered light interference, and the mask is set to 0; otherwise, it is set to 1.

[0082] ;

[0083] In the formula, Represents a binary mask for thermal inertial gating. Represents the spatial coordinates of the feature map, with 0 indicating a suppressed position (judged as illumination interference / unreliable) and 1 indicating a retained position (the change conforms to thermal inertia / reliable). Represents the characteristic time series rate of change. Indicates the thermal inertia threshold. This indicates otherwise, meaning the condition > is not met.

[0084] Finally, the feature decoupling module 300 integrates spatial thermal saliency and temporal thermal inertia to output the final purified visible light features. Before performing the calculation, the system uses a scalar mask. and Copy and expand along the channel dimension so that its dimension matches the feature vector. Consistent, the calculation process is achieved through a gating formula:

[0085] ;

[0086] In the formula, This indicates the purified visible light characteristics of the final output. Represents the spatial coordinates of the feature map. This represents the component of the visible light feature that is orthogonal to the infrared feature. This refers to a thermally inertial gated mask. This indicates a thermally significant mask.

[0087] The gating formula expresses a dual-constraint logic: visible light features are only allowed to be retained if they satisfy three conditions: mathematical orthogonality (does not interfere with calorific value), physical conformity to thermal inertia (not light and shadow noise), and they are not in the core heat source region (does not block high-temperature points). This mechanism ensures the output features... It contains only clean background textures that are helpful for positioning and harmless.

[0088] See attached document Figure 5 The image reconstruction module 400 specifically includes a gradient field calculation unit, an anisotropic diffusion fusion unit, and a decoding reconstruction unit. In this embodiment, the image reconstruction module 400 is connected to the feature mapping module 200 and the feature decoupling module 300, and its function is to purify the visible light features carrying rich texture information. Embedded into infrared features in a non-invasive manner The image is then processed and restored to a high-resolution fused image.

[0089] In this embodiment, the gradient field calculation unit is configured to calculate the spatial geometric distribution of infrared features and establish a gradient vector field that can characterize the thermal profile of an object. The system first processes the infrared feature tensor... After taking the absolute value, channel compression is performed (such as taking the channel mean or maximum value) to obtain the infrared intensity map. The horizontal gradient is then calculated using either the Sobel operator or the Scharr operator. gradient in the vertical direction For any pixel location in the image Infrared temperature gradient vector Defined as:

[0090] ;

[0091] In the formula, Indicates position The gradient vector at that point, and These represent the gradient values ​​at that location in the horizontal and vertical directions, respectively. This represents the compressed infrared intensity map. This represents the horizontal Sobel operator convolution kernel. This represents the Sobel operator convolution kernel in the vertical direction. The gradient vector g points in the direction of the most drastic temperature change (i.e., the direction across the edge).

[0092] The anisotropic diffusion fusion unit constructs an isotherm tangential vector field based on the gradient vector field and performs constrained feature fusion. The physical principle is that in image processing, the gradient direction represents the normal direction of the edge; smoothing along this direction blurs the edge. Conversely, the tangential direction represents the isotherm direction; smoothing along this direction fills the texture without destroying the edge structure. To avoid the blurring of infrared thermal edges caused by direct superposition of visible light textures, this embodiment introduces a geometric constraint strategy of fusion along the isotherm. The anisotropic diffusion fusion unit constructs the tangential vector field by orthogonally rotating and normalizing the gradient vector. :

[0093] ;

[0094] in, Indicates position The normalized tangent vector at that point, and The horizontal and vertical gradient values ​​obtained from the aforementioned calculations are... Orthogonal rotation of gradient vector The obtained tangential direction, To prevent the use of tiny constants (e.g., 10⁻⁸) with zero denominators, this tangent vector... It indicates the direction in which the temperature value remains constant within a local area.

[0095] Based on the constructed tangential vector field, the anisotropic diffusion fusion unit utilizes the anisotropic diffusion equation to analyze the purified visible light characteristics. Guided filtering is performed. The process is mathematically modeled as a controlled heat conduction process, whose diffusion tensor... Constructed to allow information only along the tangent vector Propagation. Diffusion tensor The symmetric positive definite matrix is ​​constructed using the following formula:

[0096] ;

[0097] In the formula, Indicates position The diffusion tensor matrix at that location, This represents the diffusion coefficient along the tangential direction, and its set value is relatively large (e.g., 1.0) to allow the texture to flow along the edges. This represents the diffusion coefficient along the gradient direction, and its set value is extremely small (e.g., 0.01) to suppress texture crossing edges. For the above tangent vector, For the gradient vector mentioned above, This represents the matrix transpose operation. This represents the square of the magnitude of the gradient vector. To prevent division by zero of constants.

[0098] To implement the diffusion process in a computer system, this embodiment transforms the continuous partial differential equations into discrete iterative update steps. The anisotropic diffusion fusion unit executes... Iteration (e.g.) =3 to 5 iterations), the update formula for each iteration is as follows:

[0099] ;

[0100] In the formula, Indicates the first In the next iteration, at position The feature values ​​at the location (for multi-channel features, each channel is processed independently). Indicates the first Updated value after the next iteration This represents the discrete time step (e.g., 0.1). Represents pixels The set of four neighboring pixels: top, bottom, left, and right. express The pixel coordinates of a neighbor in the set. Representing neighboring pixels The diffusion tensor at that location, Representing neighboring pixels The eigenvalues ​​at that location. The discretization formula is physically equivalent to treating the visible light characteristics as a fluid in the diffusion tensor. Flow in a defined anisotropic medium.

[0101] Final fusion feature tensor It is formed by superimposing infrared features and diffused visible light features:

[0102] ;

[0103] In the formula, Represents the fused feature tensor. Represents the original infrared feature tensor. Indicates the process Visible light feature tensor after one iteration of anisotropic diffusion This represents the fusion intensity adjustment factor. These are trainable parameters, with initial values ​​set between 0.1 and 0.3, and are automatically optimized based on the loss function during network backpropagation training.

[0104] The decoding and reconstruction unit is responsible for fusing the feature tensor. Mapping back to pixel space employs a multi-layered cascaded upsampling structure, with each stage containing a 3×3 convolutional layer, a batch normalization layer, a ReLU activation function, and an upsampling layer. The upsampling layers use bilinear interpolation or transposed convolution to progressively restore the feature map resolution to the original image size. Finally, a convolutional layer with 1 or 3 output channels (depending on whether the output is a grayscale fusion image or a pseudo-color fusion image) is used to generate the final infrared-visible multi-scale fused image. .

[0105] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0106] To verify the practical application effect and robustness of this system under complex working conditions, this embodiment selects fault diagnosis of high-voltage substation equipment in power inspection as the application scenario. This application scenario requires both accurate measurement of the temperature of overheated spots (infrared characteristics) and clear reading of equipment nameplates and surface textures (visible light characteristics), and faces strong outdoor light interference, making it highly representative.

[0107] The binocular vision acquisition terminal used in this embodiment is configured as follows:

[0108] Infrared thermal imager: Employs an uncooled vanadium oxide focal plane detector with a resolution of 640×512, thermal sensitivity (NETD) <30mK, wavelength response range of 7.5~14μm, and outputs 14-bit radiation temperature data.

[0109] Visible light camera: It uses a 1 / 1.8-inch CMOS sensor with a resolution of 1920×1080 and is equipped with a low-distortion fixed-focus lens.

[0110] Computing unit: Embedded edge computing module (21TOPS computing power), which deploys the fusion algorithm model of this invention.

[0111] The experimental subjects were disconnecting switch contacts and transformer bushings in a 220kV substation. Data acquisition covered two time periods: noon (strong sunlight, high background thermal radiation) and evening (weak sunlight) to test the system's thermal inertia gating and anisotropic diffusion performance.

[0112] Multimodal data acquisition and correction: The system captures infrared images of the disconnector switch contacts. With visible light images Because the field of view (FOV) of infrared lenses differs from that of visible light lenses, the data preprocessing module 100 is based on a pre-calibrated homography matrix. The visible light image was resampled to a resolution of 640×512 and pixel-level registered with the infrared image. At this point, the overheated contact in the infrared image and the contact contour in the visible light image were aligned in coordinates. Strict alignment is required.

[0113] Dual-stream feature extraction: The feature mapping module 200 processes data in parallel through a dual-stream network, and the infrared branch extracts low-frequency thermal features characterizing the temperature gradient. Low-frequency thermal characteristics The diffusion range of the high-temperature area (fault point) is clearly delineated; the visible light branch extracts high-frequency detail features including metallic luster, nameplate text, and bolt texture. .

[0114] Feature decoupling and illumination interference suppression, orthogonal decoupling: calculation exist Projecting along the direction removes contour lines parallel to the thermal profile in visible light, retaining orthogonal pure texture components (such as dirt marks and cracks):

[0115] Thermal inertial gating: The system detects a bright area on the surface of the porcelain bottle in three consecutive frames ( Significant intensity flickering occurred within 0.3 s (caused by cloud cover or slight changes in viewing angle). The temporal rate of change of the region was calculated. (Preset thermal inertia threshold) If the system determines that the change does not conform to the physical heat conduction law, it will automatically generate a suppression mask to filter out visible light artifacts in the area and prevent them from interfering with thermal diagnosis.

[0116] Anisotropic diffusion reconstruction: Image reconstruction module 400 calculates the isothermal tangential field of infrared features. In the high-temperature core region of the contact, the temperature gradient is large, and the tangent vector strictly follows the isotherm direction. The module controls the visible light texture features after purification. Only along the tangential direction flow.

[0117] Results: The bolt edge texture in visible light is smoothly embedded into the infrared thermal image, but does not cross the temperature gradient direction, thus avoiding blurring of the boundaries of high-temperature points. The final fused image output shows both the contact temperature as high as 85°C (pseudo-color encoding) and clearly displays the rust texture on the contact surface and the phase markings on the sides.

[0118] Three existing mainstream fusion algorithms were selected as comparison examples:

[0119] Comparative Example 1 (Traditional Laplace Pyramid Fusion): Based on multi-scale decomposition, the high-frequency coefficients are fused by taking the largest absolute value, and the low-frequency coefficients are fused by taking the average value.

[0120] Comparative Example 2 (General Fusion Based on CNN): An end-to-end convolutional neural network without physical constraints is used to directly train feature reconstruction through the L2 loss function.

[0121] Example (method of the present invention): Enable feature orthogonal decoupling and anisotropic diffusion constraint.

[0122] Information entropy (EN): measures the amount of information contained in an image; the higher the value, the better.

[0123] Average gradient (AG): measures the sharpness and texture detail of an image; a higher value is better.

[0124] Root mean square error of temperature (Temp-RMSE): Measures the temperature deviation of the fused image relative to the original infrared thermography data; a lower value is better. This is the most critical indicator for power detection, requiring that the original temperature value not be altered during the fusion process.

[0125] The experimental data are shown in the table below:

[0126] Measurement method Information entropy (EN) Average gradient (AG) Root mean square error of temperature (Temp-RMSE) Overall evaluation Comparative Example 1 6.82 4.15 2.85℃ The texture is clear, but it produces a severe halo effect at high temperature edges, resulting in lower temperature readings at fault points and a high false alarm rate. Comparative Example 2 7.15 5.23 1.92℃ The visual effects are good, but due to the lack of physical constraints, the visible light shadows are incorrectly merged into low-temperature areas, which disrupts the thermal field distribution. Example 7.48 6.12 0.34℃ While significantly improving texture clarity, it has minimal temperature error and almost completely preserves the original infrared radiation characteristics.

[0127] See attached document Figure 6 The figure shows a quantitative comparison of the above three indicators between Comparative Example 1 (traditional Laplacian pyramid fusion), Comparative Example 2 (CNN-based general fusion), and this embodiment. The horizontal axis of the figure represents different experimental groups, and the vertical axis represents the indicator values ​​(the unit for the RMSE indicator is ℃).

[0128] Image detail representation (information entropy and average gradient): such as Figure 6 As shown in the light gray histogram (information entropy) and dark gray histogram (average gradient), the method in this embodiment achieves optimal values ​​in both metrics. Specifically, the information entropy of this embodiment reaches 7.48, and the average gradient reaches 6.12, significantly higher than Comparative Example 1 (EN: 6.82, AG: 4.15) and Comparative Example 2 (EN: 7.15, AG: 5.23). This indicates that the feature decoupling module described in this invention effectively removes and preserves high-frequency texture features (such as nameplate characters and equipment corrosion marks) in visible light images, resulting in a clearer texture and richer information in the fused image, overcoming the problem of detail loss in traditional methods.

[0129] Thermal data fidelity (root mean square error of temperature): such as Figure 6 The black bar chart (root mean square error of temperature) shows a negative indicator. Comparative Examples 1 and 2 have RMSE values ​​of 2.85℃ and 1.92℃ respectively, both exhibiting significant temperature measurement deviations, which are insufficient to meet the ±1℃ accuracy requirements of industrial scenarios such as power line inspection. In contrast, the RMSE value of this embodiment is only 0.34℃, representing a significant order-of-magnitude reduction in error.

[0130] Based on the experimental data above, the infrared-visible multi-scale fusion system provided by this invention improves the detail representation of images by more than 20% while ensuring extremely low temperature error (RMSE < 0.4℃). This solution effectively resolves the contradiction between unclear equipment texture and inaccurate equipment temperature measurement in existing technologies, verifying the advancement and practicality of the feature decoupling and anisotropic diffusion-based technical approach in complex industrial monitoring scenarios.

[0131] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An infrared-visible multi-scale fusion system based on deep feature extraction, characterized in that, include: The data preprocessing module is used to establish the geometric mapping relationship between infrared images and visible light images, perform pixel-level spatial registration and signal standardization processing, and generate input tensors in a unified format that are strictly aligned in the spatiotemporal dimension. The feature mapping module, connected to the data preprocessing module, uses a dual-stream network architecture to extract the deep semantic features of infrared and visible light images respectively, and maps the heterogeneous features to a unified feature vector space through channel projection operation. The feature decoupling module, connected to the feature mapping module, is used to perform feature orthogonalization decomposition in the feature vector space, separate and remove parallel components that are linearly related to infrared features in the visible light features, retain orthogonal texture components, detect the temporal dynamic changes of the texture components, suppress transient interference based on physical thermal inertia logic, and output purified visible light features. The image reconstruction module, connected to the feature mapping module and the feature decoupling module, is used to analyze the spatial gradient distribution of infrared features, construct a guiding vector field perpendicular to the temperature gradient direction, constrain the purified visible light features in the infrared features to undergo anisotropic diffusion fusion only along the direction of the guiding vector field, and reconstruct a multi-scale fused image.

2. The infrared-visible multi-scale fusion system based on depth feature extraction according to claim 1, characterized in that, The data preprocessing module includes a parameter calibration unit and a spatial registration unit: The parameter calibration unit is configured to acquire the intrinsic parameter matrices, distortion coefficients, rotation matrix and translation vector between the infrared thermal imager and the visible light camera; The spatial registration unit is configured to construct a homography matrix based on the intrinsic parameter matrix, the rotation matrix, the translation vector, and the reference plane normal vector, perform distortion correction processing on the original image using the distortion coefficients, and perform perspective transformation interpolation on the distortion-corrected visible light image according to the homography matrix so that it coincides with the field of view of the infrared image.

3. The infrared-visible multi-scale fusion system based on depth feature extraction according to claim 1, characterized in that, The data preprocessing module further includes a data tensor quantization unit for performing the following processing: For infrared images, their radiation intensity data are linearly mapped to a normalized interval and used as the infrared input tensor; For visible light images, the color space is converted from RGB to YCbCr, and the luminance channel data is extracted as the visible light input tensor.

4. The infrared-visible multi-scale fusion system based on depth feature extraction according to claim 1, characterized in that, The dual-stream neural network in the feature mapping module includes an infrared feature extraction branch and a visible light feature extraction branch. The infrared feature extraction branch and the visible light feature extraction branch maintain symmetry in network structure, but do not share weight parameters during training, so as to learn thermal radiation feature representation and reflected light texture feature representation respectively.

5. The infrared-visible multi-scale fusion system based on depth feature extraction according to claim 1, characterized in that, The feature mapping module further includes a dimension projection alignment unit, which is used to connect convolutional layers with a kernel size of 1 to the output ends of the infrared feature extraction branch and the visible light feature extraction branch, respectively. The infrared features and visible light features are projected to the same channel dimension through a linear weighted combination of channel dimensions. The convolutional layer is followed by a batch normalization layer instead of a nonlinear activation function to preserve the directional information of the feature vector.

6. The infrared-visible multi-scale fusion system based on depth feature extraction according to claim 1, characterized in that, The feature decoupling module includes a vector orthogonal projection unit, which is used to treat the pixels on the feature map as feature vectors, calculate the projection vector of the visible light feature vector in the direction of the infrared feature vector as the parallel component, and calculate the difference between the visible light feature vector and the parallel component by vector subtraction to obtain the orthogonal component. The orthogonal components characterize the texture structure information in the visible light image that is independent of the infrared thermal radiation distribution.

7. The infrared-visible multi-scale fusion system based on depth feature extraction according to claim 6, characterized in that, The feature decoupling module further includes a thermal saliency calculation unit, which is used to calculate the mean and variance of local neighborhood features of each pixel in the infrared image, perform a weighted summation of the mean and variance of local neighborhood features, and map the summation result through a Sigmoid activation function to generate thermal saliency weights, and perform weighted suppression on the orthogonal components based on the thermal saliency weights.

8. The infrared-visible multi-scale fusion system based on depth feature extraction according to claim 6, characterized in that, The feature decoupling module also includes a timing alignment unit and a thermal inertia gating unit: The temporal alignment unit is configured to use an optical flow algorithm to calculate the optical flow field between the current time and the previous time, and to align the feature data of the previous time to the coordinate system of the current time. The thermal inertia gating unit is used to calculate the difference magnitude between the orthogonal component at the current moment and the aligned orthogonal component at the previous moment as the time change rate. When the time change rate is greater than the preset thermal inertia threshold, the corresponding area is determined to be an illumination interference area and a suppression mask is generated; otherwise, it is determined to be an effective texture area that conforms to physical thermal inertia.

9. The infrared-visible multi-scale fusion system based on depth feature extraction according to claim 1, characterized in that, The image reconstruction module includes a gradient field calculation unit and an anisotropic diffusion fusion unit: The gradient field calculation unit is configured to use an edge detection operator to calculate the horizontal and vertical gradients of infrared features and construct a gradient vector field. The anisotropic diffusion fusion unit is configured to perform orthogonal rotation and normalization on the gradient vector in the gradient vector field to construct an isothermal tangential vector field indicating the direction in which the temperature value remains unchanged in a local region.

10. The infrared-visible multi-scale fusion system based on depth feature extraction according to claim 9, characterized in that, The anisotropic diffusion fusion unit is further configured to construct a diffusion tensor, which is formed by a linear combination of the outer product of the tangential vector of the isotherm tangential vector field and the outer product of the gradient vector of the gradient vector field, wherein the weighting coefficient corresponding to the outer product of the tangential vector is set to be greater than the weighting coefficient corresponding to the outer product of the gradient vector, so that the purified visible light features mainly diffuse along the isotherm direction. The image reconstruction module also includes a decoding reconstruction unit, which is used to superimpose infrared features with visible light features after anisotropic diffusion, and reconstruct a fused image through a decoding network containing an upsampling layer.

Images